An open-world multi-modal continual learning method and system

Abstract

The application discloses an open world multi-modal continuous learning method and system, which comprises the following steps: extracting visual features of samples in a data stream and generating pseudo-text features for fusion to obtain multi-modal fusion features; performing coarse judgment on the samples in the data stream based on an energy adaptive threshold, and verifying the pseudo-labels of the known category candidate samples based on spatial consistency verification; when the new category buffer reaches a saturation state, performing clustering operation on the samples in the new category buffer to obtain new category clustering results, registering the new category as a known category, and updating the category prototype; after the new category is registered, performing incremental update on the continuous learning model, and the incremental update adopts a hybrid anti-forgetting strategy. The application establishes a unified multi-modal feature space without relying on category names, improves the pseudo-label quality through adaptive threshold and spatial consistency verification, and effectively suppresses catastrophic forgetting through the hybrid anti-forgetting strategy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an open-world multimodal continuous learning method and system. Background Technology

[0002] Continuous generalized category discovery is a core task of open-world continuous learning. It requires the model to simultaneously classify known categories, detect and discover new categories, and dynamically expand the label space when receiving sequentially arriving unlabeled data streams, while preventing catastrophic forgetting of already learned categories.

[0003] Existing methods for continuous generalized category discovery typically pre-train a visual model on a labeled base category dataset to obtain discriminative feature representations. They then receive an unlabeled data stream, use fixed or manually tuned thresholds to divide samples into known category candidates and new category candidates, perform clustering on the new category candidates to discover new categories, and suppress catastrophic forgetting through mechanisms such as experience replay. These methods mainly rely on single-modal visual features for recognition and clustering.

[0004] However, the aforementioned existing technologies have two specific drawbacks: First, fixed thresholds lack robustness in non-stationary data streams where category distributions evolve dynamically over time, leading to systematic misclassification and contaminating the training signal. Second, single-modal visual features have limitations in distinguishing fine-grained categories that are visually similar but semantically different, and unlabeled data lacks category names, making it impossible to effectively utilize the cross-modal semantic capabilities of the visual-language model, resulting in insufficient semantic discrimination ability in the feature space. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes an open-world multimodal continuous learning method and system. This invention establishes a unified multimodal feature space without relying on category names, improves the quality of pseudo-labels through adaptive thresholds and spatial consistency verification, and effectively suppresses catastrophic forgetting through a hybrid anti-forgetting strategy.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides an open-world multimodal continuous learning method, comprising the following steps: Visual features of samples in the data stream are extracted using the visual encoder in the continuous learning model; pseudo-text features are generated based on the visual features using the text embedding synthesizer in the continuous learning model; and the visual features and pseudo-text features are fused to obtain multimodal fusion features. The data stream samples are coarsely judged based on an energy adaptive threshold. Samples with energy scores exceeding the threshold are routed to a new category buffer, while samples with energy scores below the threshold are identified as candidate samples of known categories and assigned pseudo-labels. The pseudo-labels of candidate samples of known categories are verified based on spatial consistency verification. Samples with consistency scores below the set value are rerouted to a new category buffer. When the new category buffer reaches saturation, clustering is performed on the samples in the new category buffer to obtain the new category clustering result, and the new category is registered as a known category, and the category prototype is updated. After registering a new category, incremental updates are performed on the continuous learning model. The incremental updates employ a hybrid anti-forgetting strategy, which includes experience replay, knowledge distillation, and dynamic architecture expansion.

[0007] In a further technical solution, the text embedding synthesizer learns the mapping from visual features to the frozen text encoder input, and generates pseudo-text embeddings for unlabeled samples without relying on category names; the visual features and pseudo-text embeddings are weighted and fused according to learnable weight parameters to obtain multimodal fusion features.

[0008] A further technical solution involves determining the energy adaptive threshold as follows: fitting a Gaussian distribution to the energy scores of unlabeled samples in the data stream to obtain the mean and standard deviation; determining a global baseline threshold based on the mean and standard deviation; and determining a category-specific threshold for the average energy of each category with a pseudo-label, using the category-specific threshold as the decision boundary for coarsely judging the samples.

[0009] A further technical solution is that the spatial consistency verification method is as follows: in the multimodal fusion feature space, the neighborhood of the candidate sample of the known category is divided into concentric shells, the number of labeled samples that are consistent with and inconsistent with the pseudo label in each shell is counted, the spatial consistency score is calculated, the candidate sample with a consistency score higher than a set threshold is confirmed as a known category sample and its pseudo label is retained; the candidate sample with a consistency score lower than the set threshold is rerouted to the new category buffer.

[0010] A further technical solution is that the experience replay specifically involves: based on a memory buffer storing representative samples of each known category, during incremental updates, replaying known category samples from the memory buffer, mixing them with high-confidence pseudo-label samples in the current data stream, and using them together for continuous learning model training.

[0011] A further technical solution is that the knowledge distillation is achieved by calculating the divergence loss between the softened output probability distributions of the teacher model and the student model on known categories, and the divergence loss is used to constrain the parameter updates of the student model and limit the offset of the known category response.

[0012] A further technical solution is that the dynamic architecture expansion specifically involves: in the classifier of the continuous learning model, dynamically adding corresponding classification heads for newly registered categories, while retaining the classification heads corresponding to existing categories, so that the output space of the classifier can be dynamically expanded as the number of known categories increases.

[0013] Secondly, this invention provides an open-world multimodal continuous learning system, comprising the following modules: The multimodal feature extraction and fusion module is configured to: extract visual features of samples in the data stream using the visual encoder in the continuous learning model; generate pseudo-text features based on the visual features using the text embedding synthesizer in the continuous learning model; and fuse the visual features and pseudo-text features to obtain multimodal fused features. The two-stage discrimination module is configured to: perform coarse discrimination on samples in the data stream based on an energy adaptive threshold, route samples with energy scores exceeding the threshold to a new category buffer, and identify samples with energy scores below the threshold as candidate samples of known categories and assign pseudo-labels; and verify the pseudo-labels of candidate samples of known categories based on spatial consistency verification, and reroute samples with consistency scores below a set value to a new category buffer. The new category discovery and registration module is configured to: when the new category buffer reaches saturation, perform clustering operations on the samples in the new category buffer to obtain the new category clustering results, register the new category as a known category, and update the category prototype; The hybrid anti-forgetting incremental update module is configured to perform incremental updates on the continuous learning model after a new category is registered. The incremental updates employ a hybrid anti-forgetting strategy, which includes experience replay, knowledge distillation, and dynamic architecture expansion.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention introduces a text embedding synthesizer to generate pseudo-text embeddings for unlabeled samples, establishing a unified multimodal feature space independent of category names. This allows prototype computation, confidence estimation, and clustering to all be performed within a semantically rich space, solving the technical problem of insufficient semantic discrimination ability between fine-grained categories in existing single-modal visual features. It also demonstrates stronger discriminative power for visually similar but semantically different categories. Furthermore, by dynamically estimating the decision boundary between known and new categories based on energy-based adaptive thresholds, a Gaussian distribution is fitted to the energy scores of the unlabeled data stream to obtain a global baseline threshold. For each category with pseudo-labels, a category-specific threshold is derived, avoiding systematic misclassification caused by the dynamic evolution of category distribution in non-stationary data streams due to fixed thresholds, thus significantly improving the reliability of pseudo-labels.

[0015] This invention employs a hybrid anti-forgetting strategy comprised of three complementary mechanisms: experience replay, knowledge distillation, and dynamic architecture expansion. These mechanisms provide direct supervision at the data level, constrain parameter drift at the functional level, and isolate new and old knowledge at the architectural level. This effectively suppresses catastrophic forgetting during the continuous integration of new categories, achieving a balance between stability and plasticity. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0017] Figure 1 This is a schematic diagram of the process framework of an open-world multimodal continuous learning method according to the present invention; Figure 2 This is a schematic diagram of the hybrid anti-forgetting strategy of the present invention; Figure 3 The values ​​are the known class accuracy and novel class accuracy on different datasets, where (a) is the known class accuracy on the CIFAR-100 dataset, (b) is the novel class accuracy on the CIFAR-100 dataset, (c) is the known class accuracy on the Food-101 dataset, (d) is the novel class accuracy on the Food-101 dataset, (e) is the known class accuracy on the CORe50 dataset, and (f) is the novel class accuracy on the CORe50 dataset. Figure 4 The relationships between the final step accuracy of all classes / old classes / new classes and the novelty buffer size are shown for different datasets. (a) shows the relationship between the final step accuracy of all classes / old classes / new classes and the novelty buffer size on the CIFAR-100 dataset, (b) shows the relationship between the final step accuracy of all classes / old classes / new classes and the novelty buffer size on the Food-101 dataset, and (c) shows the relationship between the final step accuracy of all classes / old classes / new classes and the novelty buffer size on the CORe50 dataset. Detailed implementation method: The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0020] Terminology Explanation: Continuous learning: refers to a learning paradigm in which a model can continuously learn new knowledge while retaining previously learned knowledge when receiving sequentially arriving data streams. Its core challenge is catastrophic forgetting, which is the sharp decline in the model's performance on old tasks when learning new tasks.

[0021] Open-world continuous learning: Building on continuous learning, this further requires the model to be able to process unlabeled data streams containing unseen categories, without needing to know the number of new categories or task boundaries in advance, and to simultaneously perform known category classification, new category detection and discovery, and dynamic expansion of the label space.

[0022] Adaptive threshold: The decision boundary is dynamically estimated based on the energy score distribution of all samples in the unlabeled data stream, including two levels: a global baseline threshold and a class-specific threshold. The global baseline threshold is obtained by fitting a Gaussian distribution to the energy scores, while the class-specific threshold is derived from the average energy of each class with pseudo-labels. Together, they replace the fixed threshold, enabling the decision boundary to be adaptively updated as the composition of the data stream changes.

[0023] Spatial Consistency: Inspired by Tobler's First Law of Geography, this mechanism assesses the reliability of pseudo-labels by examining the label distribution of local neighborhoods of candidate samples of known categories in a multimodal fusion feature space. By dividing the neighborhood into concentric shells and calculating a weighted consistency score, geometric constraints are imposed on samples with blurred boundaries, further refining pseudo-labels.

[0024] New category buffer: A storage area used to temporarily store candidate samples that have been identified as new categories during the two-stage identification process.

[0025] Memory buffer: A storage area used to store representative samples for each known class.

[0026] Catastrophic forgetting: When neural networks learn new categories, the recognition performance of previously learned categories drops sharply due to parameter updates, which is a core challenge that needs to be overcome in continuous learning.

[0027] Example 1 like Figure 1 As shown, this embodiment provides an open-world multimodal continuous learning method, including the following steps: S101: Use the visual encoder in the continuous learning model to extract visual features of samples in the data stream; use the text embedding synthesizer in the continuous learning model to generate pseudo-text features based on the visual features; fuse the visual features and pseudo-text features to obtain multimodal fusion features.

[0028] In step S101, visual features are extracted from the samples in the input data stream, and a differentiated text feature generation path is executed according to whether the sample carries a label. The visual features and pseudo-text features are fused to obtain multimodal fusion features, providing a unified feature representation for subsequent recognition, clustering and classification tasks.

[0029] Specifically: like Figure 1 As shown, a CLIP dual encoder architecture is employed. First, for each frame of image samples... Visual features are extracted using the CLIP visual encoder:

[0030] in, For visual encoders, This refers to the dimension of the visual features. In this embodiment, the visual encoder uses CLIP ViT-B / 16.

[0031] Subsequently, if the sample is a labeled sample, that is, a sample with a category label in the basic or incremental stage. Input the corresponding category name into the frozen CLIP text encoder to generate text features. ,in For the frozen CLIP text encoder, This is the dimension of the text features. However, this depends on the category name. For unlabeled new category samples, due to the lack of a category name, they can only be limited to visual features, which leads to a decrease in the performance of subsequent category prototype estimation, confidence scoring, and novelty detection.

[0032] To unlock the potential of CLIP text modalities in handling novel instances, this embodiment proposes a Text Embedding Synthesizer (TES). The TES learns a mapping from visual features to frozen CLIP text encoder input tokens. If the sample is unlabeled, meaning it's a sample in the continuous learning phase data stream without category labels, the text encoder cannot be directly invoked due to the lack of category names. Therefore, visual features must be incorporated. The input is a Text Embedding Synthesizer (TES). The TES converts visual features into a sequence of input tokens for the text encoder using a learned mapping function, and then outputs pseudo-text features via a frozen text encoder. .

[0033] Finally, the visual features and pseudo-text features are fused to obtain the multimodal fusion features:

[0034] in, These are learnable parameters. This fused feature will serve as input for all subsequent modules.

[0035] In the fusion mechanism of step S101, TES enables prototype computation, confidence estimation and clustering to be performed in a unified multimodal space, eliminating the bottleneck that new category candidate samples rely solely on visual features; and the multimodal space has stronger discriminative power for visually similar but semantically different categories; at the same time, the new category candidate samples in the buffer carry semantic priors from pseudo-text embeddings, which can generate more compact and more separable clustering structures in the unsupervised discovery process.

[0036] S102: Based on the energy adaptive threshold, perform a coarse judgment on the samples in the data stream, route the samples with energy scores exceeding the threshold to the new category buffer, and determine the samples with energy scores not exceeding the threshold as known category candidate samples and assign pseudo-labels; based on spatial consistency verification, verify the pseudo-labels of known category candidate samples, and reroute the samples with consistency scores lower than the set value to the new category buffer.

[0037] In step S102, reliable novelty detection in unlabeled data streams is a core challenge, as a single criterion is often insufficient. To address this, this embodiment proposes a two-stage discrimination method: first, a coarse separation of known and unknown samples at the distribution level is achieved using an energy-based adaptive threshold; then, a spatial consistency check is performed, further refining known candidate samples by filtering out ambiguous pseudo-labels. This collaborative mechanism improves the quality of pseudo-labels in subsequent clustering and incremental training.

[0038] Phase 1: Coarse Judgment Based on Energy Adaptive Threshold Compared to the overconfident softmax probability, the energy score calculated based on logits provides a more robust out-of-distribution metric.

[0039] For those with and temperature Each unlabeled sample Its energy fraction is calculated as follows:

[0040] in, For category The number of logits, This is a temperature parameter. It controls the scale of the energy score. A higher energy score indicates a greater likelihood that the sample belongs to an out-of-distribution category (i.e., a new class). Compared to methods based on Softmax probabilities, energy scores calculated based on the raw output values ​​are more robust to overconfidence problems.

[0041] To avoid the vulnerability introduced by a fixed threshold, all sample sets in the unlabeled data stream are processed. The energy fraction is fitted with a Gaussian distribution to obtain the mean. and standard deviation Set a global baseline threshold:

[0042] in, To control the stringency of the hyperparameters, this global baseline threshold can be adaptively updated as the composition of the data stream changes. This applies to each category with pseudo-labels. In this embodiment, its average energy is calculated. And derive the threshold for a specific category:

[0043] Use a category-specific threshold as the decision boundary for coarsely judging samples: Energy fraction satisfies Samples are considered known candidate samples and assigned pseudo-labels; while Samples that do not meet the criteria are considered as candidate samples for the new class and are directly routed to the new class buffer. This mechanism allows classes with compact feature distributions to obtain stricter thresholds, while classes with ambiguous feature distributions obtain looser boundaries, thus achieving adaptive decision boundary estimation.

[0044] Phase Two: Precise Judgment Based on Spatial Consistency Inspired by Tobler's first law of geography, this embodiment evaluates the reliability of pseudo-labels for candidate samples of known categories generated in the first stage by examining their local neighborhoods in the multimodal fusion feature space. For samples with pseudo-labels... candidate samples , and make its radius The neighborhood is divided into A concentric shell. In each shell In the middle, set For the label equals The number of labeled samples, For labels different The spatial consistency score is defined as the number of labeled samples.

[0045] in, As a distance-based attenuation weight, closer shells contribute more to the consistency score. For the first The number of labeled samples in each shell that are consistent with the pseudo-label; For the first The number of labeled samples in each shell that are inconsistent with the pseudo-label; The neighborhood radius determines the total number of concentric shell layers.

[0046] In this embodiment, the consistency threshold is set to 0.5: Candidate samples with a consistency score higher than the set threshold are dominated by labeled samples that are consistent with the pseudo-label in their local neighborhood, and are identified as known category samples and their pseudo-labels are retained, and they enter the known category training process; Candidate samples with a consistency score lower than the set threshold have ambiguous pseudo-labels and are rerouted to the new category buffer.

[0047] S103: When the new category buffer reaches saturation, perform clustering operation on the samples in the new category buffer to obtain the new category clustering result, register the new category as a known category, and update the category prototype.

[0048] In this embodiment, both stages will route the new category candidate samples to a capacity of... The buffer is used. When the buffer reaches saturation, clustering operations are performed in the multimodal feature space (using the K-means++ algorithm, which automatically determines the number of clusters based on silhouette coefficients), and pseudo-text embeddings generated by TES are used as semantic priors. A new category will be registered as a known category at the next time step, its category prototype will be calculated, and then the buffer will be cleared to continue the data stream processing operation.

[0049] Specifically: The new category of buffer has a capacity limit. When the number of candidate samples accumulated in the buffer reaches the capacity limit. When the clustering operation is triggered, it is performed on all samples in the buffer within the multimodal fusion feature space. The K-means++ algorithm, which automatically determines the number of clusters based on the silhouette coefficient, is used, eliminating the need to pre-specify the number of new categories. Clustering quality is selected based on the maximum silhouette coefficient. The value represents the number of new categories discovered in this study. During the clustering process, pseudo-text embeddings generated by a text embedding synthesizer are used as semantic priors to assist clustering, enabling visually similar but semantically different samples to be correctly grouped, resulting in a more compact and separable cluster structure.

[0050] After clustering was completed, the following were found A new category is registered as a known category at the next time step, i.e. ,in, The complete sample categories after clustering. The sample categories before clustering. This refers to the new sample categories generated through clustering.

[0051] Simultaneously, a category prototype (represented by the mean of the multimodal fusion features of all samples in that category) is calculated for each newly registered category, which is used for confidence estimation and new category detection in subsequent stages. After registration is complete, the new category buffer is cleared, and new category candidate samples are received from subsequent data streams.

[0052] S104: After registering a new category, perform incremental updates on the continuous learning model. The incremental updates employ a hybrid anti-forgetting strategy, which includes experience replay, knowledge distillation, and dynamic architecture expansion.

[0053] Whenever a new batch of categories is registered, the model is updated to accommodate them. However, this updating inevitably exposes previously learned categories to the risk of catastrophic forgetting, and no single defense mechanism can address this problem independently. Therefore, this embodiment integrates three complementary techniques into a hybrid strategy.

[0054] As shown in Figure 2, these mechanisms work at different stages of a single update: old class samples are replayed and merged with new class samples into the training batch, the frozen teacher model constrains the student model through distillation, and the classifier is expanded by dynamically adding new output heads.

[0055] Experience replay A memory buffer stores representative samples for each known class. During incremental updates, samples of known classes are replayed from the memory buffer and mixed with samples with high-confidence pseudo-labels in the current data stream to form training batches, which are then used for model training. The replayed samples of known classes provide direct supervision signals for the old classes, supplementing the shortcomings of knowledge distillation at the data level and preventing excessive drift of model parameters towards the new classes.

[0056] Knowledge distillation Before each incremental update, the current model state is frozen as the teacher model (the model in stage t-1), retaining the known class responses before the update. The student model (the model in stage t) is trained using distillation loss to align with the known classes. The softened output probability distribution on the surface. Let... and These represent the teacher model and student model for the samples, respectively. In category The logits (raw output value) at the distillation temperature The softening probability distribution of the teacher model is as follows:

[0057] in, For distillation temperature parameters, higher temperature values ​​make the probability distribution smoother, which helps to convey information about inter-class relationships. For the teacher model (stage t-1 frozen copy) sample In category The original output value; Given the current set of known categories, the summation range covers all known categories. Similarly, define the softening probability distribution for the student model. .

[0058] The distillation loss is the average KL divergence over training batch B:

[0059] in, The number of samples in the training batch; For the teacher model in the known categories The probability of softening on; For student models in known categories The softening probability on; Given a set of known categories, this distillation loss constrains the student model's output distribution on known categories to align with the teacher model, limiting the bias in responses to known categories and preventing parameter drift while learning new categories.

[0060] Dynamic architecture extension Because the label space expands continuously with the registration of new categories, a fixed-size classifier cannot represent the newly registered categories. During each incremental update, the classifier in the continuously learning model is updated to represent the newly registered categories. Dynamically add corresponding categories The system uses a new set of classification heads, while retaining all existing classification heads for each category, allowing the classifier's output space to dynamically expand as the number of known categories increases. New knowledge is incorporated into the newly added parameters, while the representational power of the old categories remains unchanged, preventing interference from new category learning on the old category classification heads at the architectural level.

[0061] In this embodiment, before entering the continuous learning phase (steps S101 to S104 are executed cyclically), this embodiment also includes a foundational phase: based on the labeled base category dataset. The continuous learning model is initially trained, where Let the category labels be the initial known set of categories, denoted as . The following tasks are completed in the foundational phase: (1) Train the visual encoder to obtain discriminative visual feature representations; (2) Calculate the category prototypes of each basic category using the visual features and text features of labeled samples, as described in step S101; (3) Train the text embedding synthesizer to learn the mapping from visual features to the text encoder input token, enabling it to generate high-quality pseudo-text embeddings for unlabeled samples. After the basic stage is completed, the continuous learning stage begins. In each incremental stage (stage 1 to T), the trained category prototypes and text embedding synthesizer are reused without retraining.

[0062] The method provided in this embodiment is verified through experiments, specifically as follows: set up Datasets: This paper evaluates four benchmark datasets covering different domains. CIFAR-10 contains 60,000 32×32 pixel images across 10 categories, with a 5 / 5 known / new category split. CIFAR-100 contains 60,000 images across 100 fine-grained categories, with an 80 / 20 split, presenting a greater challenge. Food-101 contains 101,000 real-world food images across 101 categories, with an 81 / 20 split, reflecting a long-tailed distribution. CORe50 contains 164,866 RGB images covering 10 household item categories, using an 8-known-category and 2-new-category split, capturing real-world robot perception data streams with varying backgrounds. These datasets collectively cover different levels of difficulty, scale, and granularity.

[0063] Implementation details: Visual features are extracted using CLIP ViT-B / 16 (512-dimensional). TES from GET uses K=16 text tokens and a 512-dimensional embedding. An energy threshold is pre-trained for 100 epochs (using an SGD optimizer with a learning rate of 0.1, momentum of 0.9, and weight decay of 1e-4). During data stream processing, an adaptive threshold is applied with a decay factor... Updates were performed; the consistency threshold was set to 0.5; both the memory buffer and the new class buffer held 2,000 samples. Clustering was performed using the K-means++ algorithm based on silhouette coefficients, triggered when the buffers were full. After each clustering, the model was fine-tuned for 5 epochs (learning rate 1e-3). All results are the average of three runs.

[0064] Evaluation Metrics: This embodiment employs two evaluation protocols. For the continuous learning baseline method, this embodiment reports: known accuracy (accuracy on known classes after data stream processing), forgetting rate (the decrease in accuracy after training), and overall accuracy (the joint accuracy of known classes and new classes). For comparisons of generalized class discovery, this embodiment follows the Hungarian matching algorithm and reports known accuracy and new class accuracy (based on optimal label assignment). These metrics comprehensively evaluate the model's retention ability (known accuracy, forgetting rate), discovery ability (new class accuracy), and overall performance (overall accuracy).

[0065] Baseline Methods: This embodiment compares with eight continuous learning methods covering mainstream approaches: EWC (Fisher Information-based Regularization), SI (Online Importance Evaluation), ER (Pure Empirical Replay), iCaRL (Empirical Replay with Nearest Mean Classification), DER++ (Empirical Replay with Logit Distillation), Erace (Asymmetric Cross-Entropy), LwF (Data-Free Distillation), and A-GEM (Gradient Projection). These methods are all closed-world methods, evaluating only known classes (without touching new classes), providing a more rigorous benchmark, as this embodiment must simultaneously handle the new class discovery task. Furthermore, this embodiment introduces two GCD methods specifically designed for data flow: iGCD and OCGCD, and evaluates them under the same data flow protocol.

[0066] Experimental results Comparison with continuous learning methods: Table 1 compares the performance of this embodiment with eight continuous learning baseline methods on the CIFAR-10, CIFAR-100, and Food-101 datasets. This embodiment achieves the highest known accuracy and the lowest forgetting rate on all datasets.

[0067] On the CIFAR-10 dataset, this embodiment achieved a forgetting rate of only 1.97%, less than half that of the DER++ method (5.51%), and also maintained the lowest forgetting rates on CIFAR-100 (5.55%) and Food-101 (11.44%). These performance improvements were achieved under more challenging protocols: the baseline method had prior knowledge of the number of new classes and task boundaries, while this embodiment did not.

[0068] Table 1 compares the accuracy (%) and forgetting rate (%) of the known class method with the continuous learning method. ↑ / ↓ indicate higher / lower is better. Best results are shown in bold. In the last two columns, "Number of Classes Prior" indicates whether the method knows the total number of new classes in advance, or must run without this information; "Task Boundary" indicates whether the task boundary is provided, or is inferred automatically by the model. This embodiment runs under a more stringent setting of unlabeled and task-boundary-free methods.

[0069] Table 1

[0070] This advantage stems from three complementary design features. First, energy-based adaptive thresholding dynamically re-estimates the boundaries of known / new classes at each step, ensuring reliable pseudo-labels are generated under distribution shifts, thus directly improving known accuracy. Second, spatial consistency verification filters out samples with ambiguous boundaries, which is particularly beneficial on the fine-grained Food-101 dataset. Furthermore, the pseudo-text embeddings generated by TES enrich the feature space with semantic structure, improving energy estimation and buffer clustering, creating a virtuous cycle between better feature representation and pseudo-labels.

[0071] This leads to two deeper observations. First, the forgetting rate gap between our method and DER++ widens with increasing dataset complexity (from 3.54% on CIFAR-10 to 6.92% on Food-101), indicating that semantic regularization and two-stage filtering become more critical in cases of dense semantic confusion. Second, our method outperforms baseline methods with prior knowledge without privileged information, suggesting that closed-world continuous learning benchmarks may overestimate the utility of knowing the number of new categories or task boundaries—further confirming the importance of truly open-world evaluation, which is precisely the direction this work is pursuing.

[0072] Comparison with generalized category discovery methods like Figure 3 As shown, the stepwise accuracy comparison of the method in this embodiment, OCGCD, and iGCD is presented with increasing time steps. (a)(c)(e) Known class accuracy on the CIFAR-100, Food-101, and CORe50 datasets. (b)(d)(f) Novel class accuracy on the CIFAR-100, Food-101, and CORe50 datasets.

[0073] Known accuracy within incremental time steps: This embodiment consistently outperforms iGCD and OCGCD across all datasets and time steps. On CIFAR-100, it achieves 91.63% accuracy at T1 and retains 70.42% at T5, while the baseline methods achieve only 60.87% and 62.64% respectively in the final step; similar trends are observed on Food-101 and CORe50. This advantage stems from the synergy of clean training input and a hybrid anti-forgetting mechanism: the two-stage pipeline filters out new class noise before model updates, ensuring a significant reduction in contamination of the distillation target. This clean positive supervision signal prevents the model from excessively drifting towards new classes, a failure mode exhibited by baseline methods lacking a dedicated filtering stage.

[0074] New class accuracy within incremental time steps: All methods exhibit a characteristic peak at T1, followed by a sharp decline and plateau. Our method achieves 62.36% on CIFAR-100 and 65.97% on Food-101 at T1, significantly higher than the two baseline methods, before declining and stabilizing. The peak at T1 reflects ideal initial conditions (clean pre-training, well-calibrated thresholds, empty buffers), while the subsequent decline reflects accumulating difficulties: the energy distribution of known classes widens, the feature space becomes crowded, and the scale of the Hungarian assignment problem increases. Performance degradation is widespread, but our method maintains a significant advantage at every step. On CIFAR-100, our method outperforms iGCD and OCGCD by 36.00 and 28.98 percentage points, respectively, at T1, and remains the highest at T5 (21.82% vs. 16.86% and 15.75%). The larger absolute decrease in performance (40.54 vs. 9.39 and 17.86) is a direct result of the higher starting point of this method—the baseline method starts close to its lower bound. It also maintains the same ranking on Food-101 and CORe50. The advantage of this embodiment's method lies not in escaping the inherent performance degradation, but in maintaining the highest new class accuracy across the entire data stream. This trajectory highlights the fundamental paradox: elevating discovered classes to a known state gradually weakens the model's calibration capabilities. This embodiment mitigates this problem through pseudo-text embeddings generated by TES, which provide a semantically structured feature space that preserves discriminative power as the number of classes increases, resulting in a smoother performance degradation than the baseline method—a subtle difference masked by simply looking at the absolute decrease metric.

[0075] Sensitivity to the size of the new category buffer: such as Figure 4As shown, this embodiment scans the new class buffer size near the default value (which controls the balance between discovery frequency and sample sufficiency per invocation). On CIFAR-100 and Food-101, performance remains almost flat (overall accuracy changes are within 2 and 3 percentage points, respectively; the default value achieves optimal overall accuracy), confirming the model's robustness over a wide range when the data stream is long enough. On CORe50, performance monotonically increases within its feasible range, but this reflects a higher discovery frequency (smaller buffers trigger more events) rather than an improvement in quality; new class accuracy remains stable, with only old class accuracy changing. This paper sets the maximum capacity to 3,000, which still triggers sufficient discovery events on short CORe50 data streams. Essentially, the buffer acts as a control knob balancing frequency and sufficiency; robustness increases with data stream length, and the default setting in this embodiment is precisely at the performance optimum. This insensitivity means that practitioners can choose the buffer size based on computational constraints as long as the data stream is long enough without substantially impacting the final performance. This robustness also indicates that the clustering quality is primarily governed by the semantic richness of the TES-guided feature space, rather than by cumulative scheduling, which validates the effectiveness of our proposed multimodal backbone network design.

[0076] ablation experiment To quantify the independent contribution of each core component, ablation experiments were conducted on three datasets, evaluating four variants: (1) without adaptive thresholding, replacing the energy-based adaptive threshold with a fixed threshold; (2) without spatial consistency, removing the spatial consistency verification stage; (3) without anti-forgetting mechanism, disabling the hybrid anti-forgetting mechanism; and (4) without multimodal fusion, replacing the multimodal (vision-language) backbone network with a unimodal visual encoder. The summarized results are shown in Table 2.

[0077] Table 2 shows the ablation experiments of the proposed GCDHAF method on three datasets. The best results are shown in bold.

[0078] Table 2

[0079] Ablation analysis: The complete model consistently outperforms all variants across all datasets, confirming that the contributions of each component are not redundant. Ablation experiments reveal a clear hierarchy of component importance and their synergistic interactions.

[0080] Adaptive threshold: Replacing it with a fixed threshold caused the forgetting rate on the Food-101 dataset to increase from 11.44% to 21.29%, and reduced the known accuracy and overall accuracy on all datasets. In non-stationary data streams, static boundaries accumulate systematic misclassification errors, thus contaminating the training signal and the distillation target. This cumulative effect makes adaptive capability a prerequisite for stable model operation.

[0081] Spatial Consistency: Removing this geometric filtering mechanism results in a 12.85 percentage point drop in overall accuracy on the CIFAR-100 dataset. Those blurry samples that barely pass the energy threshold enter the training process with unreliable pseudo-labels, which is particularly detrimental in fine-grained scenarios, as small errors can significantly deviate from the decision boundary. Therefore, this stage acts as a quality control mechanism to prevent the model from training on uncertainty, and its importance increases with the refinement of class granularity.

[0082] Multimodal fusion: Using only visual features causes the known accuracy on the CIFAR-100 dataset to plummet from 81.76% to 55.83%, and on the Food-101 dataset from 77.52% to 45.15%, while the forgetting rate increases significantly. On the CIFAR-10 dataset, however, this performance degradation is much milder (from 91.53% to 81.98%), because the coarse-grained categories on this dataset can be distinguished solely by visual features. This dataset-dependent performance degradation indicates that multimodal fusion provides crucial semantic structure when visual cues are insufficient. Without TES embeddings, the feature space becomes disorganized, leading to entanglement between semantically different but visually similar categories. This entanglement simultaneously reduces the accuracy of energy estimation, the quality of pseudo-labels, and the effectiveness of buffer clustering, confirming that the fusion mechanism is the foundation upon which other components rely; these other components are merely downstream optimization techniques and cannot compensate for the deficiencies caused by impoverished feature representations.

[0083] Hybrid anti-forgetting mechanism: Disabling this mechanism leads to a significant increase in the forgetting rate and the most severe drop in known accuracy across all settings. Catastrophic forgetting not only erodes historical accuracy but also distorts the feature space structure as parameters drift towards new categories, thereby impairing threshold estimation and pseudo-label quality in subsequent steps, creating a vicious feedback loop. This highlights that an anti-forgetting mechanism is a systemic requirement, maintaining the integrity of the feature representations needed by all other modules.

[0084] In all ablation experiments, the impact of missing components on the CIFAR-10 dataset was negligible (overall accuracy remained at 95%), but the impact on the CIFAR-100 and Food-101 datasets was extremely severe. Each component becomes increasingly critical as the number of classes and inter-class similarity increases, reflecting typical real-world deployment scenarios. This progressive sensitivity indicates a synergistic rather than a simple additive effect among the components: adaptive thresholding ensures temporal adaptability, spatial consistency enforces geometric coherence, multimodal fusion provides semantic structure, and anti-forgetting mechanisms maintain stability. The failure of any single component leads to a disproportionate performance drop on challenging datasets, suggesting that the overall effect is greater than the sum of its parts—a hallmark of a well-designed open-world continuous learning system.

[0085] Example 2 This embodiment provides an open-world multimodal continuous learning system, including the following modules: The multimodal feature extraction and fusion module is configured to: extract visual features of samples in the data stream using the visual encoder in the continuous learning model; generate pseudo-text features based on the visual features using the text embedding synthesizer in the continuous learning model; and fuse the visual features and pseudo-text features to obtain multimodal fused features. The two-stage discrimination module is configured to: perform coarse discrimination on samples in the data stream based on an energy adaptive threshold, route samples with energy scores exceeding the threshold to a new category buffer, and identify samples with energy scores below the threshold as candidate samples of known categories and assign pseudo-labels; and verify the pseudo-labels of candidate samples of known categories based on spatial consistency verification, and reroute samples with consistency scores below a set value to a new category buffer. The new category discovery and registration module is configured to: when the new category buffer reaches saturation, perform clustering operations on the samples in the new category buffer to obtain the new category clustering results, register the new category as a known category, and update the category prototype; The hybrid anti-forgetting incremental update module is configured to perform incremental updates on the continuous learning model after a new category is registered. The incremental updates employ a hybrid anti-forgetting strategy, which includes experience replay, knowledge distillation, and dynamic architecture expansion.

[0086] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0087] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An open-world multimodal continuous learning method, characterized in that, include: Visual features of samples in a data stream are extracted using a visual encoder in a continuous learning model. The text embedding synthesizer in the continuous learning model generates pseudo-text features based on visual features; the visual features and pseudo-text features are then fused to obtain multimodal fusion features. Based on the energy adaptive threshold, samples in the data stream are coarsely judged. Samples with energy scores exceeding the threshold are routed to the new category buffer, and samples with energy scores not exceeding the threshold are identified as candidate samples of known categories and assigned pseudo-labels. Based on spatial consistency verification, the pseudo-labels of candidate samples of known categories are verified, and samples with consistency scores lower than the set value are rerouted to the new category buffer. When the new category buffer reaches saturation, clustering is performed on the samples in the new category buffer to obtain the new category clustering result, and the new category is registered as a known category, and the category prototype is updated. After registering a new category, incremental updates are performed on the continuous learning model. The incremental updates employ a hybrid anti-forgetting strategy, which includes experience replay, knowledge distillation, and dynamic architecture expansion.

2. The open-world multimodal continuous learning method as described in claim 1, characterized in that, The text embedding synthesizer learns the mapping from visual features to the frozen text encoder input, generating pseudo-text embeddings for unlabeled samples without relying on category names; the visual features and pseudo-text embeddings are weighted and fused according to learnable weight parameters to obtain multimodal fusion features.

3. The open-world multimodal continuous learning method as described in claim 1, characterized in that, The energy adaptive threshold is determined as follows: a Gaussian distribution is fitted to the energy scores of unlabeled samples in the data stream to obtain the mean and standard deviation, and a global baseline threshold is determined based on the mean and standard deviation; and a category-specific threshold is determined for the average energy of each category with a pseudo-label, and the category-specific threshold is used as the decision boundary for coarsely judging the samples.

4. The open-world multimodal continuous learning method as described in claim 1, characterized in that, The spatial consistency verification method is as follows: in the multimodal fusion feature space, the neighborhood of the candidate sample of the known category is divided into concentric shells, the number of labeled samples that are consistent with and inconsistent with the pseudo label in each shell is counted, the spatial consistency score is calculated, and the candidate sample with the consistency score higher than the set threshold is confirmed as a known category sample and its pseudo label is retained. Candidate samples with a consistency score below a set threshold are rerouted to a new category buffer.

5. The open-world multimodal continuous learning method as described in claim 1, characterized in that, The experience replay specifically involves: based on a memory buffer storing representative samples of each known category, during incremental updates, replaying known category samples from the memory buffer and mixing them with high-confidence pseudo-label samples in the current data stream for continuous learning model training.

6. The open-world multimodal continuous learning method as described in claim 1, characterized in that, The knowledge distillation is achieved by calculating the divergence loss between the softened output probability distributions of the teacher model and the student model on known categories. The divergence loss constrains the parameter updates of the student model and limits the offset of the known category response.

7. The open-world multimodal continuous learning method as described in claim 1, characterized in that, The dynamic architecture expansion specifically involves dynamically adding corresponding classification heads to the classifier of the continuous learning model for newly registered categories, while retaining the classification heads corresponding to existing categories, so that the output space of the classifier can be dynamically expanded as the number of known categories increases.

8. An open-world multimodal continuous learning system, characterized in that, Includes the following modules: The multimodal feature extraction and fusion module is configured to: extract visual features of samples in the data stream using the visual encoder in the continuous learning model; generate pseudo-text features based on the visual features using the text embedding synthesizer in the continuous learning model; and fuse the visual features and pseudo-text features to obtain multimodal fused features. The two-stage discrimination module is configured to: perform a coarse judgment on samples in the data stream based on an energy adaptive threshold, route samples with energy scores exceeding the threshold to a new category buffer, and determine samples with energy scores not exceeding the threshold as candidate samples of known categories and assign pseudo-labels; Based on spatial consistency verification, the pseudo-labels of candidate samples of known categories are verified, and samples with consistency scores lower than the set value are rerouted to the new category buffer. The new category discovery and registration module is configured to: when the new category buffer reaches saturation, perform clustering operations on the samples in the new category buffer to obtain the new category clustering results, register the new category as a known category, and update the category prototype; The hybrid anti-forgetting incremental update module is configured to perform incremental updates on the continuous learning model after a new category is registered. The incremental updates employ a hybrid anti-forgetting strategy, which includes experience replay, knowledge distillation, and dynamic architecture expansion.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps of an open-world multimodal continuous learning method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the open-world multimodal continuous learning method as described in any one of claims 1-7.

Images