Visual target clustering method based on long and short term multi-modal memory pool

Abstract

The invention discloses a visual target clustering method based on a long and short term multi-modal memory pool. The method comprises the following steps: S1, extracting initial visual features and initial semantic description from an image containing a visual target as initial bimodal feature data; s2, performing matching calculation on the initial bimodal feature data of the visual target and existing target memory pools to obtain a matching degree score of the visual target and each existing target memory pool; s3, judging whether the visual target and a visual target corresponding to the target memory pool are the same object according to the matching degree score, if so, updating the target memory pool, and if not, creating a new target memory pool corresponding to the visual target; s4, precipitating and filtering the updated target memory pool, and converting the initial bimodal characteristic data into stable bimodal characteristic data; and S5, for a newly input image containing the visual target, repeatedly executing S1 to S4, and generating a visual target clustering result.

Description

Technical Field

[0001] This invention relates to the field of visual analysis technology, and in particular to a visual target clustering method based on long short-term multimodal memory pools. Background Technology

[0002] With the widespread deployment of security monitoring systems and the explosive growth of streaming media applications such as live streaming and short videos, video data has become the most prevalent core content format in the current internet field. The emergence of massive amounts of video content has created an urgent demand for downstream applications such as intelligent video analysis and target memory retrieval. Among these, the effective clustering of key semantic elements such as people, vehicles, and objects is a core prerequisite for the implementation of these applications. The quality of the clustering directly determines the usability and accuracy of applications such as target retrieval, precise querying, and intelligent judgment.

[0003] Currently, the mainstream technical solutions in the field of visual object clustering are based on traditional ReID technology (Re-Identification, which covers sub-fields such as pedestrian re-identification, vehicle re-identification, and entity re-identification). The core implementation logic of this type of technology is: to perform instance clustering calculations solely based on image feature vectors (Embedding), while simultaneously using a fixed list of structured attributes (such as common attributes like age, gender, and height) to assist in object retrieval.

[0004] However, in practical application scenarios such as security monitoring, smart retail, and streaming video that require long-term span tracking and clustering, the above-mentioned traditional technical solutions have significant defects and are difficult to meet the actual application needs. They have the following problems: (1) Feature drift problem. Affected by environmental and target state fluctuation factors such as changes in light intensity, temporary occlusion of objects, and changes in target posture, the visual features of the same visual target in different video frames are quite different. Especially for key targets such as personnel, temporary short-term appearance changes such as wearing clothes and hats are more frequent, which makes the single visual feature vector on which traditional ReID technology relies very easy to fail, and thus incorrectly cluster the same target into multiple independent IDs, significantly reducing the intra-instance accuracy of clustering and increasing the instance splitting degree; (2) Lack of specific semantic information for the target and insufficient fusion of text and visual features. Traditional ReID technology is mostly based on CNN (convolutional neural network) or ViT (visual Transformer) to extract pixel-level visual features and simply superimposes a fixed structured attribute list to carry out retrieval. This kind of fixed attribute list has extremely poor flexibility. It uses a uniform attribute dimension for all targets and cannot specifically characterize the exclusive features of different targets, making it difficult to cope with the retrieval deviation caused by short-term appearance interference. Even though some solutions introduce multimodal large-scale VQA (visual question answering) methods to assist semantic extraction, they still use a uniform template for interaction and do not carry out personalized semantic optimization for specific targets.

[0005] In recent years, the Vision Large Language Model (VLM) technology has developed rapidly. Its core advantage lies in its ability to accurately mine rich semantic information of target objects, achieving deep integration of visual features and textual semantics. Based on this, how to leverage the semantic mining capabilities of VLM models to overcome the shortcomings of traditional technical solutions and improve the accuracy and stability of visual target clustering has become a core technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] In view of the above-mentioned prior art, the present invention provides a visual target clustering method based on long short-term multimodal memory pool, which mainly solves the technical problems existing in the background art.

[0007] To achieve the above objectives, the technical solution of this invention is implemented as follows: This invention discloses a visual target clustering method based on a long short-term multimodal memory pool. The clustering method includes the following steps: S1. Extract initial visual features and initial semantic descriptions from images containing visual targets as initial bimodal feature data; S2. The initial bimodal feature data of the visual target is matched with the existing target memory pool to obtain the matching score between the visual target and each existing target memory pool. S3. Based on the matching score, determine whether the visual target and the visual target corresponding to the target memory pool are the same object. If they are the same object, update the target memory pool. If they are not the same object, create a new target memory pool corresponding to the visual target. S4. Perform sedimentation and filtering on the updated target memory pool to transform the initial bimodal feature data into stable bimodal feature data; S5. For a new input image containing visual targets, repeat steps S1 to S4 to generate visual target clustering results.

[0008] Preferably, the target memory pool consists of a short-term memory pool and a long-term memory pool. The short-term memory pool is used to store the initial visual feature set and the initial semantic description set of the corresponding visual target in order to capture the temporary state changes of the visual target. The long-term memory pool is used to store a stable set of visual feature vectors and a stable set of semantic description texts for visual targets, so as to achieve interference-resistant recognition of the visual targets.

[0009] Preferably, the initial visual features and initial semantic descriptions corresponding to the visual target are extracted, specifically including: For an input image containing a visual target, the target subject region is extracted using a target detection algorithm; For the separated target subject region, a visual feature extraction model is used to perform feature parsing and output an initial visual feature vector that uniquely corresponds to the visual target. For the separated target subject region, a visual language multimodal model combined with a customized set of prompt words is used to generate an initial semantic description text that uniquely corresponds to the visual target.

[0010] Preferably, the visual target is matched with each existing target memory pool to achieve a matching degree score, specifically including: For each existing memory pool, the initial visual features of the visual target are calculated. The cosine distance between the image visual feature vectors in the short-term memory pool and the image visual feature vectors in the long-term memory pool, and the initial semantic description of the visual target are calculated. The cosine distance between the text description group in the short-term memory pool and the text description group in the long-term memory pool; Based on the maximum distance in both the visual and semantic dimensions, calculate the maximum difference distance between the visual target and the short-term memory pool in each existing memory pool; Based on the minimum distance in both the visual and semantic dimensions, calculate the minimum difference distance between the visual target and the long-term memory pool in each existing memory pool; The matching score between the visual target and each existing memory pool is obtained based on the maximum and minimum difference distances.

[0011] Preferably, determining whether the visual target and the visual target corresponding to the target memory pool are the same object based on the matching score specifically includes: selecting the maximum value from all matching scores, and comparing the maximum value of the selected matching score with a preset threshold. If the maximum value of the selected matching score is less than the preset threshold, then the visual target and the visual target corresponding to the target memory pool corresponding to the maximum value of the matching score are the same object. If the maximum value of the selected matching score is greater than the preset threshold, then the visual object is a new visual object.

[0012] Preferably, if the visual object is the same object in the existing target memory pool, the initial bimodal feature data of the visual object is updated to the short-term memory pool of the target memory pool according to a preset strategy, replacing the oldest stale feature stored in the short-term memory pool.

[0013] Preferably, for the updated existing target memory pool, all visual features in its long-term memory pool and short-term memory pool are clustered, and the obtained visual feature clusters are used as visual features in the long-term memory pool to achieve the deposition of visual features.

[0014] Preferably, for the updated existing target memory pool, all initial semantic descriptions in its short-term memory pool are aggregated using a large language model to obtain an initial aggregation result. Based on this initial aggregation result, the large language model is used again to filter and summarize the existing stable semantic descriptions in the long-term memory pool. The output of the large language model is used as the stable semantic description text in the long-term memory pool.

[0015] Preferably, if the visual object is a new visual object, a new target memory pool is created for it. The short-term memory pool in the new target memory pool is initially the initial bimodal feature data of the visual object, and the long-term memory pool in the new target memory pool is initially an empty set.

[0016] The beneficial effects of this invention are as follows: This invention employs a multimodal feature fusion scheme combining visual and semantic features, overcoming the limitations of traditional ReID technology that relies solely on a single visual feature vector. By accurately characterizing the core attributes of the target through semantic features, the clustering process exhibits good robustness to drastic changes in appearance, such as "changing clothes" or "occlusion," significantly improving the stability of clustered targets. This effectively avoids the problem of the same target being incorrectly split into multiple IDs due to the failure of a single visual feature, ensuring the in-instance accuracy of clustering. Simultaneously, this invention adopts a long short-term memory pool collaborative strategy. By capturing temporary feature changes of the target in the short-term memory pool and storing stable features in the long-term memory pool, it achieves dynamic management and accumulation of target features. This enables adaptation to long-term, continuous appearance changes of the target, making it particularly suitable for long-term video analysis scenarios using a single camera. This solves the shortcomings of traditional technologies, which lack human-like adaptive "memory" and cannot adjust clustering strategies based on historical features. Building upon this foundation, this invention achieves a matching score by calculating the geometric mean of the "most similar" distance and the "least similar" distance. This strategy accurately balances the short-term fluctuations and long-term stable attributes of the target, further improving the accuracy of matching and clustering, reducing instance splitting, and providing a high-quality clustering foundation for downstream applications such as target retrieval and precise querying. Furthermore, this invention can flexibly adapt to various model combinations, allowing for the selection of suitable models based on the computational power and accuracy requirements of the actual application scenario. This significantly enhances the applicability and implementation flexibility of the solution, better meeting the visual target clustering needs of different scenarios such as security monitoring, smart retail, and streaming video. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the visual target clustering method based on long short-term multimodal memory pools in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the implementation process of an embodiment of this application. Detailed Implementation

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. In the following description, the expression "some embodiments" refers to a subset of all possible embodiments; however, it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0020] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0021] It should be understood that the present invention can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Furthermore, the terminology used herein is intended only to describe particular embodiments and is not intended to limit the invention. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “compose” and / or “comprising,” when used in this specification, identify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0022] It should also be noted that when an element is referred to as being "fixed to" another element, it can be directly attached to the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "inner," "outer," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementation.

[0023] To fully understand this invention, a detailed structure will be presented in the following description to illustrate the technical solution proposed by this invention. Optional embodiments of the invention are described in detail below; however, in addition to these detailed descriptions, the invention may have other embodiments.

[0024] Please refer to the attached document. Figures 1 to 2 This application provides a visual target clustering method based on a long short-term multimodal memory pool. The clustering method includes the following steps: S1. Extract initial visual features and initial semantic descriptions from images containing visual targets as initial bimodal feature data.

[0025] In some implementations, the initial visual features and initial semantic descriptions corresponding to the visual target are extracted, specifically including: For an input image containing a visual target, the target subject region is extracted using a target detection algorithm; For the separated target subject region, a visual feature extraction model is used to perform feature parsing and output an initial visual feature vector that uniquely corresponds to the visual target. For the separated target subject region, a visual language multimodal model combined with a customized set of prompt words is used to generate an initial semantic description text that uniquely corresponds to the visual target.

[0026] Preferably, the aforementioned images containing visual targets can be derived from independent, standalone images, such as static images triggered by snapshots in security monitoring scenarios or snapshots of target objects uploaded by users; or they can be video images extracted frame by frame from continuous video streams, such as single-frame images extracted at a preset frame rate or on demand in scenarios like real-time monitoring videos, traffic condition videos, and streaming media videos.

[0027] Regardless of the image source, visual targets in the image need to be located using an object detection algorithm to extract the target subject region. In some optional implementations, the object detection algorithm can be selected from mainstream object detection algorithms such as YOLO series algorithms (e.g., YOLOv5, YOLOv7, YOLOv8), Faster R-CNN, SSD (Single Shot MultiBoxDetector), RetinaNet, and CenterNet. These algorithms have the characteristics of strong real-time performance, high detection accuracy, and good adaptability to multi-scale targets. They can quickly and accurately separate the target subject from the background, effectively remove redundant background information in the image, avoid background interference affecting the accuracy of subsequent feature extraction, and ensure that the feature extraction process focuses only on the visual target itself.

[0028] For the separated target subject region, a visual feature extraction model is used for targeted feature analysis. In some optional implementations, the visual feature extraction model can be selected from convolutional neural network (CNN) models (such as ResNet, EfficientNet, MobileNet), visual Transformer (ViT) models, visual branches of CLIP models, Swing Transformers, etc. Through in-depth mining of visual information such as appearance details, contour structure, and texture features of the target subject, the model outputs an initial visual feature vector that uniquely corresponds to the visual target. This vector can accurately represent the visual essential attributes of the target and provide a reliable basis for subsequent visual dimension matching calculations.

[0029] For the same target subject region, a visual language multimodal model is used, combined with a customized set of prompt words to carry out semantic mining. In some optional implementations, the visual language multimodal model can be selected from models with cross-modal semantic understanding capabilities such as CLIP (Contrastive Language-Image Pre-training), BLIP (Bootstrapping Language-Image Pre-training), BLIP-2, Flava, and LLaVA (Large Language and Vision Assistant). The customized set of prompt words can guide the model to focus on the core features and unique attributes of the visual target for semantic description, avoiding the semantic ambiguity caused by generalized descriptions. Finally, an initial semantic description text uniquely corresponding to the visual target is generated, comprehensively capturing the describable semantic features of the target and providing effective support for semantic dimension matching calculation.

[0030] Its customized prompts can include, for example: "Describe the prominent features of the scene and provide some keywords!", "What are the people in the scene doing?", "Describe the appearance of the people in the scene?", etc.

[0031] S2. The initial bimodal feature data of the visual target is matched with the existing target memory pool to obtain the matching score between the visual target and each existing target memory pool. Specifically, the target memory pool described in this application is a short-term memory pool (STM) and a long-term memory pool (LTM). Each memory pool stores a set of image visual feature vectors and a corresponding set of text descriptions for a specific visual target. For example, the short-term memory pool is used to store the initial set of visual features and the initial set of semantic descriptions for the corresponding visual target in order to capture the temporary state changes of the visual target. The long-term memory pool is used to store the stable set of visual feature vectors and the stable set of semantic description texts for the visual target in order to achieve anti-interference recognition of the visual target.

[0032] After extracting the visual target, a matching score is performed between the visual target and each existing target memory pool. This specifically includes: For each existing memory pool, the initial visual features of the visual target are calculated. The cosine distance between the image visual feature vectors in the short-term memory pool and the image visual feature vectors in the long-term memory pool, and the initial semantic description of the visual target are calculated. The cosine distance between the text description group in the short-term memory pool and the text description group in the long-term memory pool; Based on the maximum distance in both the visual and semantic dimensions, the maximum difference distance between the visual target and the short-term memory pool in each existing memory pool is calculated using the following formula:

[0033] in, For the weight of the visual dimension, The maximum distance in the visual dimension. This represents the maximum distance in the semantic dimension.

[0034] Based on the minimum distances in both the visual and semantic dimensions, the minimum difference distance between the visual target and the long-term memory pool in each existing memory pool is calculated using the following formula:

[0035] in, The minimum distance in the visual dimension. This represents the minimum distance in the semantic dimension.

[0036] The matching score between the visual target and each existing memory pool is obtained based on the maximum and minimum difference distances. The calculation formula is as follows:

[0037] in Score the match.

[0038] Suppose the new input visual target to be clustered is a new frame image of pedestrian A, from which the initial visual features are extracted. and initial semantic description There are two memory pools: memory pool 1 corresponds to pedestrian A, and memory pool 2 corresponds to pedestrian B. For memory pool 1, calculate the initial visual features of pedestrian A. The cosine distance between each visual feature vector and the visual feature vector group of the image in short-term memory pool 1 is calculated, and the maximum value of multiple calculation results is selected as the maximum distance in the visual dimension. Similarly, the initial semantic description of pedestrian A is calculated. The cosine distance between the visual target and each text description vector in the short-term memory pool of memory pool 1 is calculated, and the maximum value of multiple calculation results is selected as the maximum semantic dimension distance. Based on a preset calculation formula, the maximum difference distance between the visual target and each existing short-term memory pool is calculated. .

[0039] Furthermore, the initial visual features of pedestrian A are calculated. The cosine distance between each visual feature vector and the image visual feature vector group in the long-term memory pool of memory pool 1 is calculated, and the minimum value of multiple calculation results is selected as the minimum visual dimension distance in the long-term memory pool. Similarly, the initial semantic description of pedestrian A is calculated. The cosine distance between the visual target and each text description vector in the long-term memory pool of memory pool 1 is calculated, and the minimum value of multiple calculation results is selected as the semantic dimension minimum distance in the long-term memory pool. Based on a preset calculation formula, the maximum difference distance between the visual target and the long-term memory pool of each existing memory pool is calculated. .

[0040] Based on the aforementioned matching degree calculation formula, the matching degree score between pedestrian A and memory pool 1 can be obtained. Similarly, the matching degree score between pedestrian A and memory pools 2 / 3 / 4 can be obtained through the aforementioned process.

[0041] S3. Based on the matching score, determine whether the visual target and the visual target corresponding to the target memory pool are the same object. If they are the same object, update the target memory pool. If they are not the same object, create a new target memory pool corresponding to the visual target. In some optional implementations, determining whether the visual target and the visual target corresponding to the target memory pool are the same object based on the matching score specifically includes: selecting the maximum value from all matching scores, and comparing the maximum value of the selected matching score with a preset threshold. If the maximum value of the selected matching score is less than the preset threshold, then the visual target and the visual target corresponding to the target memory pool corresponding to the maximum value of the matching score are the same object. If the maximum value of the selected matching score is greater than the preset threshold, then the visual object is a new visual object.

[0042] Specifically, during the judgment process, the maximum value is selected from all the calculated matching scores. The target memory pool corresponding to this maximum value is the candidate memory pool that is most similar to the visual target to be clustered in the existing memory pool. The operation of selecting the maximum value can effectively lock the optimal matching object and avoid the judgment bias caused by the confusion of multiple memory pool scores.

[0043] The maximum matching score is compared with a preset threshold, which is a boundary criterion obtained through extensive experimental calibration. For example, based on the accuracy requirements of security monitoring scenarios, the preset threshold is 0.5.

[0044] Suppose the visual target to be clustered is a new frame image of pedestrian A. Its matching score with two existing memory pools has been calculated, with memory pool 1 having a score of 0.75 and memory pool 2 having a score of 0.058. The maximum value is 0.75. This maximum value is compared with a preset threshold of 0.5. Since 0.75 is greater than the preset threshold, according to the determination rule of this embodiment, the visual target to be clustered is determined to be a new visual object. In another hypothetical scenario, the target to be clustered is a new frame image of pedestrian B. The calculated matching score with memory pool 2 is 0.4, and the scores of other memory pools are all lower than 0.4. The maximum value for filtering is 0.4. Since 0.4 is less than the preset threshold of 0.5, it is determined that the visual target to be clustered and the visual target corresponding to memory pool 2 are the same object.

[0045] Furthermore, if the visual object is the same object in the existing target memory pool, the initial bimodal feature data of the visual object is updated to the short-term memory pool of the target memory pool according to a preset strategy, replacing the oldest stale feature stored in the short-term memory pool.

[0046] Specifically, the preset strategy described in this embodiment refers to the First-In-First-Out (FIFO) strategy. The core logic of this strategy is to determine the data eviction priority according to the storage time order of data elements. That is, the earliest data element stored in the storage space is deleted first when storage resources need to be released. The data elements stored later are arranged in chronological order, always following the "first in, first out" management rule, to ensure that the data in the storage space always maintains the latest time window characteristics. For example, suppose the preset storage capacity of the short-term memory pool is 5 sets of features. The pool currently stores 5 sets of initial bimodal feature data of visual targets in chronological order, with corresponding storage timestamps t1, t2, t3, t4, and t5, where t1 is the earliest time. When the initial bimodal feature data (timestamp t6) of newly acquired visual targets needs to be stored, since the number of features in the pool has reached the capacity limit, the earliest timestamp t1 feature is deleted first according to the first-in-first-out strategy. Then, the new feature at time t6 is added to the tail of the queue. The timestamps corresponding to the updated features in the short-term memory pool are t2, t3, t4, t5, and t6, always keeping the latest 5 sets of recent temporary features.

[0047] Furthermore, if the visual object is a new visual object, a new target memory pool is created for it. The short-term memory pool in the new target memory pool is initially the initial bimodal feature data of the visual object, and the long-term memory pool in the new target memory pool is initially an empty set.

[0048] Specifically, the construction of the new target memory pool follows the rule of "initial storage in the short-term memory pool and initial emptiness in the long-term memory pool." Taking pedestrian C, who appears for the first time in a security monitoring scenario, as an example, the initial bimodal feature data of pedestrian C is stored in the short-term memory pool of the new memory pool as the initial feature data of the short-term memory pool. The long-term memory pool is initialized to an empty set because only a set of initial features of the new target has been obtained at this time, and sufficient data has not yet been accumulated to refine stable features. It is necessary to wait for the continuous input and feature updates of the target image to gradually build a stable feature set through a sedimentation and filtering process. After the new target memory pool is created, it will automatically and dynamically integrate into the existing target memory pool set and participate in the matching calculation of subsequent new visual objects, forming a complete memory pool system covering all clustered targets.

[0049] By updating the short-term memory pool using a first-in-first-out (FIFO) strategy, the short-term nature and timeliness of the short-term memory pool are effectively guaranteed. This avoids the capture bias of the recent temporary state of the target caused by the accumulation of outdated features, ensuring that the short-term memory pool can accurately reflect the latest state changes of the target and provide a reliable short-term feature benchmark for subsequent matching calculations. Meanwhile, the targeted creation of the new target memory pool enables the seamless dynamic expansion of clusters without the need to pre-set the number of clusters. This adapts to the need for the random appearance of new visual targets in real-world scenarios and solves the pain point of traditional clustering methods being unable to cope with dynamically added targets.

[0050] S4. Perform sedimentation and filtering on the updated target memory pool to transform the initial bimodal feature data into stable bimodal feature data; In some implementations, for an updated existing target memory pool, all visual features in its long-term memory pool and short-term memory pool are clustered, and the obtained visual feature clusters are used as visual features in the long-term memory pool to achieve the deposition of visual features.

[0051] Specifically, after the short-term memory pool completes feature updates, a sedimentation and filtering process is automatically triggered. The sedimentation process targets visual features, acquiring all visual feature vectors in the updated short-term memory pool and the original stable visual feature vectors in the long-term memory pool. These two types of features are integrated into a unified visual feature set. Subsequently, an unsupervised clustering algorithm (such as the KMeans algorithm) is used to cluster this set, with a preset number of cluster centers K. After clustering, k feature clusters are obtained. The core feature vector of the feature cluster containing the core attributes is selected as the updated visual features of the long-term memory pool, thus completing the sedimentation of visual features.

[0052] Taking the target memory pool corresponding to pedestrian A in a security monitoring scenario as an example, after updating the memory pool, all visual feature vectors in the short-term memory pool (assuming 5 sets of features from time t2 to t6, corresponding to pedestrian A's temporary states of wearing a hat, looking down, turning to the side, standing normally, and holding a document) and the original stable visual feature vectors in the long-term memory pool (1 set, corresponding to the core visual attributes of "adult male facial contour + medium height") are used to form a unified visual feature set containing 6 sets of features. This set is then clustered using an unsupervised clustering algorithm, where the number of cluster centers K=3 is a preset fixed value. After clustering, 3 visual feature clusters are obtained. The features of visual feature cluster 1 focus on the core attribute of "facial contour + medium height" that does not change with posture or temporary appearance. The features of visual feature cluster 2 mainly reflect posture differences (looking down, turning to the side, etc.). The features of visual feature cluster 3 include interference information such as temporary occlusion (holding a document) and wearing a hat. Based on the core attributes, the cluster center of visual feature cluster 1 is selected as the updated long-term memory pool visual feature, completing the deposition of visual features.

[0053] Furthermore, for the updated existing target memory pool, all initial semantic descriptions in its short-term memory pool are aggregated using a large language model to obtain an initial aggregation result. Based on this initial aggregation result, the large language model is used again to filter and summarize the existing stable semantic descriptions in the long-term memory pool. The output of the large language model is used as the stable semantic description text in the long-term memory pool.

[0054] Specifically, the semantic feature deposition process relies on a large language model (such as CLIP, BLIP-2, etc., which have strong semantic understanding capabilities) and is executed in two steps. First, all independently stored initial semantic description texts in the short-term memory pool are collected. This text set is input into the large language model, and combined with customized prompt words, the initial aggregation result is output. Then, the original stable semantic description texts in the long-term memory pool are extracted and input into the same large language model along with the initial aggregation result. Based on preset prompt word instructions, the original stable semantic descriptions are filtered, and the filtered results are used as the updated stable semantic description texts in the long-term memory pool. It is ensured that the number of semantic description texts in the long-term memory pool remains at a preset fixed value, thus completing the semantic feature deposition. Taking pedestrian A's memory pool as an example, firstly, all independently stored initial semantic description texts in the short-term memory pool are collected, such as "wearing a black baseball cap and a black shirt", "looking down and wearing a black shirt", "turning to the side and wearing a light-colored coat", "standing normally and wearing a black shirt", "holding a document and wearing a black shirt", etc. This set of texts is input into the large language model, combined with customized prompts, such as "summarize the core stable attributes of the target in the following descriptions and remove descriptions related to temporary states", etc., and the initial aggregation result is output. The initial aggregation result is, for example, "adult male, medium height, mole on face, often wears a black shirt". Then, the original stable semantic description text in the long-term memory pool, such as "adult male, medium height, mole on face", is extracted and input into the large language model along with the initial aggregation result. Based on the prompt instruction of "retaining stable attributes consistent with the aggregation result, removing temporary attributes, and supplementing the core stable features that have not been missed", the original stable semantic description is filtered. For example, the stable semantic description of "adult male, medium height, mole on face" is retained, and the high-frequency stable attribute "often wears black top" is added in combination with the initial aggregation result. Finally, "adult male, medium height, mole on face, often wears black top" is output as the updated stable semantic description text in the long-term memory pool.

[0055] S5. For a new input image containing visual targets, repeat steps S1 to S4 to generate visual target clustering results.

[0056] By continuously repeating the S1 to S4 process, the Long Short-Term Memory (LSM) pool for each visual target can be continuously accumulated and optimized with the input of new image data. The stable bimodal features stored in LTM have higher accuracy and effectively resist feature drift caused by factors such as changes in illumination, pose adjustment, temporary occlusion, and clothing changes, ensuring that the same target always belongs to the same cluster and avoiding ID splitting. On the other hand, by dynamically creating new memory pools, the clustering system can be expanded imperceptibly, overcoming the limitation of traditional clustering methods in dealing with incremental targets. At the same time, the synergistic effect of multimodal feature fusion and LSM pool is continuously strengthened in the loop. The addition of semantic features makes up for the lack of specific semantics in traditional ReID technology, while the capture of temporary states by STM and the anchoring of stable features by LTM make the clustering results both adaptable in real time and have cross-temporal and spatial anti-interference capabilities. Ultimately, the visual target clustering results generated through this iterative process achieve the core requirements of high accuracy within instances and low instance splitting, providing high-quality data support for subsequent applications such as target retrieval, querying, and intelligent analysis. This significantly improves the practicality and reliability of the solution in complex scenarios such as security monitoring, streaming media analysis, and smart retail.

[0057] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual target clustering method based on long short-term multimodal memory pool, characterized in that, Clustering methods include the following steps: S1. Extract initial visual features and initial semantic descriptions from images containing visual targets as initial bimodal feature data; S2. The initial bimodal feature data of the visual target is matched with the existing target memory pool to obtain the matching score between the visual target and each existing target memory pool. S3. Based on the matching score, determine whether the visual target and the visual target corresponding to the target memory pool are the same object. If they are the same object, update the target memory pool. If they are not the same object, create a new target memory pool corresponding to the visual target. S4. Perform sedimentation and filtering on the updated target memory pool to transform the initial bimodal feature data into stable bimodal feature data; S5. For a new input image containing visual targets, repeat steps S1 to S4 to generate visual target clustering results.

2. The visual target clustering method based on long short-term multimodal memory pool according to claim 1, characterized in that, The target memory pool consists of a short-term memory pool and a long-term memory pool. The short-term memory pool is used to store the initial visual feature set and the initial semantic description set of the corresponding visual target in order to capture the temporary state changes of the visual target. The long-term memory pool is used to store a stable set of visual feature vectors and a stable set of semantic description texts for visual targets, so as to achieve interference-resistant recognition of the visual targets.

3. The visual target clustering method based on long short-term multimodal memory pool according to claim 1, characterized in that, Extracting initial visual features and initial semantic descriptions corresponding to the visual target, specifically including: For an input image containing a visual target, the target subject region is extracted using a target detection algorithm; For the separated target subject region, a visual feature extraction model is used to perform feature parsing and output an initial visual feature vector that uniquely corresponds to the visual target. For the separated target subject region, a visual language multimodal model combined with a customized set of prompt words is used to generate an initial semantic description text that uniquely corresponds to the visual target.

4. The visual target clustering method based on long short-term multimodal memory pool according to claim 3, characterized in that, The visual target is then compared with existing target memory pools to determine its matching degree, specifically including: For each existing memory pool, the initial visual features of the visual target are calculated. The cosine distance between the image visual feature vectors in the short-term memory pool and the image visual feature vectors in the long-term memory pool, and the initial semantic description of the visual target are calculated. The cosine distance between the text description group in the short-term memory pool and the text description group in the long-term memory pool; Based on the maximum distance in both the visual and semantic dimensions, calculate the maximum difference distance between the visual target and the short-term memory pool in each existing memory pool; Based on the minimum distance in both the visual and semantic dimensions, calculate the minimum difference distance between the visual target and the long-term memory pool in each existing memory pool; The matching score between the visual target and each existing memory pool is obtained based on the maximum and minimum difference distances.

5. The visual target clustering method based on long short-term multimodal memory pool according to claim 4, characterized in that, Based on the matching score, it is determined whether the visual target and the visual target corresponding to the target memory pool are the same object. Specifically, this includes: selecting the maximum value from all matching scores, and comparing the maximum value of the selected matching score with a preset threshold. If the maximum value of the selected matching score is less than the preset threshold, then the visual target and the visual target corresponding to the target memory pool corresponding to the maximum value of the matching score are the same object. If the maximum value of the selected matching score is greater than the preset threshold, then the visual object is a new visual object.

6. The visual target clustering method based on long short-term multimodal memory pool according to claim 5, characterized in that, If the visual object is the same object in the existing target memory pool, then the initial bimodal feature data of the visual object is updated to the short-term memory pool of the target memory pool according to a preset strategy, replacing the oldest stale feature stored in the short-term memory pool.

7. The visual target clustering method based on long short-term multimodal memory pool according to claim 6, characterized in that, For the updated existing target memory pool, all visual features in its long-term memory pool and short-term memory pool are clustered, and the obtained visual feature clusters are used as visual features in the long-term memory pool to achieve the deposition of visual features.

8. The visual target clustering method based on long short-term multimodal memory pool according to claim 7, characterized in that, For the updated target memory pool, all initial semantic descriptions in its short-term memory pool are aggregated using a large language model to obtain an initial aggregation result. Based on this initial aggregation result, the large language model is used again to filter and summarize the existing stable semantic descriptions in the long-term memory pool. The output of the large language model is used as the stable semantic description text in the long-term memory pool.

9. A visual target clustering method based on a long short-term multimodal memory pool according to claim 6, characterized in that, If the visual object is a new visual object, a new target memory pool is created for it. The short-term memory pool in the new target memory pool is initially the initial bimodal feature data of the visual object, and the long-term memory pool in the new target memory pool is initially an empty set.

Images