Incremental implicit refined image classification method based on hyperbolic hierarchical modeling and two-way knowledge transmission

Abstract

The invention discloses an incremental implicit refinement image classification method based on hyperbolic hierarchical modeling and two-way knowledge transmission, and the method comprises the steps: 1, carrying out the incremental task division of a data set: reconstructing an original image data set into a data set with a two-stage hierarchical structure, and carrying out the division of each task data set; step 2, network training: inputting a training set of a current task, training the hyperbolic hierarchical modeling and knowledge two-way transmission network, and updating model parameters; step 3, testing and outputting classification accuracy: testing on a test set of all the seen tasks and outputting an average increment classification accuracy; and repeating the steps 2 and 3 until no new task appears. According to the method, the dynamic hierarchical relationship is continuously constructed in the hyperbolic space, and the dynamic hierarchical relationship is used as a transmission cost to guide bidirectional optimal transmission between new and old knowledge, so that the problems of poor effect and insufficient knowledge utilization of a static hard hierarchical relationship adopted by a current method are solved, and the accuracy of incremental implicit refined image classification is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to an incremental implicit refinement image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transmission, and is suitable for incremental implicit refinement image classification technology in the field of computer vision. BACKGROUND

[0002] With the development of Internet technology, the image data of the real world presents the characteristics of mass and dynamic streaming input. However, due to the reasons such as the inability to access historical data, the fixed number of output classifications and the fixed amount of parameters of the deployed model, the traditional image classification model is prone to catastrophic forgetting of old knowledge when learning new classes, and loses the ability to distinguish old classes. Therefore, the class incremental learning paradigm of machine learning is proposed and has been widely studied. The core goal is to enable the model to retain old knowledge while learning new knowledge, and to continuously learn from streaming data. However, the traditional class incremental image classification still has certain limitations. In terms of adapting to real-world scenarios, the traditional paradigm usually assumes that the labels of the image ontology will not change between tasks, that is, the new and old classes are completely independent (for example: old class "dog" - new class "cat"). However, in some more complex real-world scenarios, the labels of the image ontology often change in granularity between tasks, that is, there is a hierarchical semantic relationship between the new and old classes. Due to insufficient knowledge base, high cost of complete multi-label annotation, and other reasons, the real hierarchical relationship between classes is usually unknown, and the training images can only have single-granularity labels corresponding to the current task (for example: an image with ontology "Doberman" has single labels with different granularities in the previous and subsequent tasks, that is, old class "dog" - new class "Doberman"). In terms of modeling the human continuous learning process, cognitive science shows that humans not only incrementally learn knowledge, but also refine their understanding of knowledge by exploring the hierarchical relationship between knowledge to ensure the coherence of learned knowledge. However, the implicit knowledge refinement mechanism of humans has not been effectively modeled by the traditional paradigm.

[0003] In order to make up for the above shortcomings of the traditional paradigm, researchers propose an extended paradigm of class incremental image classification - incremental implicit refinement image classification. This paradigm simulates the real human cognitive process, incrementally trains the model based on sequential tasks, and enables the model to refine its understanding of the image ontology by exploring the implicit hierarchical relationship between classes while incrementally accumulating knowledge. The specific settings of incremental implicit refinement image classification are as follows: Figure 1As shown: the incremental implicit refinement image classification task requires the model to incrementally learn a sequence of image classification tasks, where the labels of any two tasks do not overlap, but there may be a coarse-fine-grained hierarchical relationship, and follow the general cognition that large categories are easier to learn, coarse labels (parent class) will appear in earlier tasks than corresponding fine labels (child class), so the model only holds a single granularity label of the ontology during training of each task, and the model predicts all granularity labels of the ontology seen up to the current task during inference. The incomplete information (single granularity) training condition and complete information (multi-granularity) prediction requirement of the above makes the incremental implicit refinement image classification face the following core challenges: (a) the need to incrementally mine the implicit parent-child hierarchical relationship between classes during the learning sequence task; (b) resist the catastrophic forgetting of this relationship.

[0004] In recent years, thanks to the rapid development of deep learning, a large number of class-incremental image classification methods have been proposed, which can be roughly divided into three categories: regularization-based methods, replay-based methods and architecture-based methods. The regularization-based method resists forgetting by limiting the change of model parameters across tasks, such as EWC selects key parameters and consolidates their consistency across tasks, iCaRL and LUCIR save old knowledge from old models through knowledge distillation; The replay-based method uses various strategies to replay the information of the old task for the new model, such as ER directly saves part of the original old data, AGEM saves gradient information related to old data, and IL2A uses a generative model to generate a large amount of fake old data; The architecture-based method isolates model parameters for different tasks, such as DER dynamically expands the model to add task-specific parameters for new tasks, and PackNet selects model parameters from a fixed super-network for new tasks.

[0005] Since incremental implicit refinement image classification is an extension of the incremental image classification paradigm, most of the methods mentioned above for traditional paradigms can be easily adapted to the settings of incremental implicit refinement image classification, such as ER, iCaRL, and LUCIR. However, these transfer methods ignore the special inter-class hierarchical relationships in this scenario, resulting in unsatisfactory performance. Several targeted solutions for incremental implicit refinement image classification have emerged. HCV determines hard hierarchical relationships before each task training, adds pseudo-parent class labels to subclasses based on these relationships during training, and performs prediction corrections based on these relationships during inference. UAHR proposes a hierarchical distribution alignment strategy, which also determines hard hierarchical relationships before the task and guides knowledge distillation based on these relationships during training. MTKD employs dual-teacher knowledge distillation, with one teacher distilling parent class knowledge from the initial task model and the other distilling old knowledge from the previous model, and designs a top-k prediction constraint to post-process the prediction results. MPHL designs a dual-head architecture to learn different latent spaces for parent and child classes and performs multi-granularity predictions in a cascaded manner.

[0006] Existing methods for solving incremental implicit refinement image classification have the following two drawbacks: (a) The static hard hierarchical relationships used are poor. Most methods, such as HCV, UAHR, and MPHL, use static hard hierarchical relationships determined before the task as prior knowledge to guide the training process, which may propagate and reinforce incorrect relationships; (b) Knowledge utilization is insufficient. Some methods, such as UAHR and MTKD, use the traditional knowledge distillation strategy to transfer knowledge unidirectionally from old to new, which fails to make full use of the prior knowledge contained in the association between old and new classes, especially between parent and child classes.

[0007] In view of the challenges faced by incremental implicit thinning image classification and the shortcomings of existing methods, this invention proposes a hyperbolic hierarchical modeling and bidirectional knowledge transfer network to complete the incremental implicit thinning image classification task. The hyperbolic hierarchical modeling leverages the ability of hyperbolic geometry to express tree-like hierarchical structures, periodically modeling dynamic soft hierarchical relationships in hyperbolic space. This ensures better accuracy of hierarchical relationships. Guided by the constructed hierarchical relationships, bidirectional knowledge transfer realizes forward inheritance from old to new and backward preservation from new to old knowledge through forward and backward transfer processes, respectively. This allows for more full utilization of learned knowledge and further strengthens the inter-class hierarchical associations required for incremental implicit thinning image classification. Summary of the Invention

[0008] This invention proposes an incremental implicit refinement image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transfer. The key idea of ​​this method is to continuously construct dynamic hierarchical relationships in hyperbolic space and use them as a transfer cost to guide the optimal bidirectional transfer of knowledge between old and new classes. This process establishes a feedback loop: the refined hierarchical relationships guide knowledge transfer, while the transfer benefits in turn affect the refinement of the hierarchical relationships. Hyperbolic hierarchical modeling is implemented through a dynamic hyperbolic hierarchical module, which fully utilizes the expressive power of hyperbolic geometry for hierarchy, embedding category features into the hyperbolic space to measure similarity. The resulting hyperbolic hierarchical relationships are dynamically updated after each knowledge transfer, thus benefiting from knowledge interaction. Bidirectional knowledge transfer is jointly implemented through a forward transfer module and a backward transfer module. Both modules use the current hierarchical relationship as the transfer cost. The forward transfer module transfers knowledge from the old class classifier to generate an auxiliary new class classifier, allowing the new class to inherit and effectively utilize the old knowledge. The backward transfer module transfers knowledge from the new class logit (the output of the new class classifier) ​​to generate an auxiliary old class logit, allowing the memorization of old knowledge to benefit from the learning of the new class. Furthermore, this invention proposes forward and backward transmission losses to further guide the two transmission processes, maximizing the benefits of each function. The main technical problems addressed by this method include the poor performance of static hard-level relationships and insufficient knowledge utilization in current methods, thereby improving the accuracy of incremental implicit refinement image classification.

[0009] The technical solution adopted in this invention is an incremental implicit thinning image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transfer, which includes the following steps:

[0010] Step 1, Incremental task partitioning of the dataset: Reconstruct the original image dataset into a dataset with a two-level hierarchical structure, and partition the dataset for each task.

[0011] Step 2, Training the Network: Input the training set of the current task, train the hyperbolic hierarchical modeling and bidirectional knowledge transfer network, and update the model parameters;

[0012] Step 3, Test and output classification accuracy: Test on the test set of all known tasks and output an average incremental classification accuracy;

[0013] Repeat steps 2 and 3 until no new tasks appear.

[0014] Furthermore, step 1, which involves incrementally dividing the dataset into tasks, specifically includes the following:

[0015] To simulate the scenario of incremental implicit refinement in image classification, the original image classification dataset is first reconstructed into a two-level hierarchical dataset containing parent and child classes. Then, according to a certain task division setting, each category is distributed in each task of the incremental task sequence to obtain the final training and test sets for each task.

[0016] Four datasets with a two-level hierarchical structure were used: IIRC-CIFAR, IIRC-ImageNet-Full, IIRC-ImageNet-Lite, and IIRC-ImageNet-Subset. These datasets were reconstructed from two commonly used open-source image classification datasets, CIFAR100 and ImageNet, with the latter two being lightweight versions of IIRC-ImageNet-Full. For each hierarchical dataset, the parent class's data was sampled from each of its corresponding child classes and then merged. The data for each child class consisted of 80% of its original data, meaning that the parent class and each of its child classes shared 20% of the data. Specifically, the IIRC-CIFAR dataset has 15 parent classes and 100 child classes, with each parent class corresponding to 4-8 child classes. It is divided into 22 tasks, with the initial task containing 10 parent classes and each subsequent task containing 5 classes. The IIRC-ImageNet-Full dataset has 85 parent classes and 998 child classes, with each parent class corresponding to 3-118 child classes. It is divided into 35 tasks, with the initial task containing 63 parent classes and each subsequent task containing 30 classes. The IIRC-ImageNet-Lite dataset has 69 parent classes and 264 child classes, and it is divided into 10 tasks, with the initial task containing 63 parent classes and each subsequent task containing 30 classes. The IIRC-ImageNet-Subset dataset has 10 parent classes and 100 child classes, and it is divided into 11 tasks. Unlike the previous three datasets, each task in this dataset has 10 classes, and the 10 parent classes are evenly distributed across the first ten tasks.

[0017] Furthermore, the training network in step 2 includes the following:

[0018] For each task, the training set for the current task is input to train the established hyperbolic hierarchical modeling and bidirectional knowledge transfer network, and the model parameters are updated. The training process consists of the following 5 steps:

[0019] 1) Input image data to obtain model output;

[0020] Given a task sequence The t-th task The training set is defined as in These are the image set and the label set, respectively. The labels for different tasks do not overlap. These are the i-th input image sample and its corresponding single label, The number of samples in the training set; the replay set is defined as Its old task Sampled from the training data, and compared with the training set Participate in the current mission together Learning to resist catastrophic forgetting. Remembering the current task. The number of category labels is remember The total number of category tags is Training task At that time, firstly, each image data Input the old model Θ respectively t-1 =(Φ t-1 Ψ t-1 ) and the new model Θ t =(Φ t ,Ψ t This will allow you to obtain some of the necessary output results, including... For feature extractors, For classifiers, Ψ t It can be divided into two parts: the old class classifier. And new class classifier On the one hand, x is input to the old model Θ from the previous task, whose parameters are now frozen. t-1 Obtain the old model classifier Ψ t-1 Output: Old Model On the other hand, x is the input to the new model Θ for the current task. t Obtain the new model feature extractor Φ t Output: Euclidean features The old classifier for the new model Output: Old Class and new class classifiers Output: New class

[0021] 2) Construct a hierarchical relationship matrix using a dynamic hyperbolic hierarchical module;

[0022] The training set obtained afterwards and replay set The Euclidean features z of all samples are input into the dynamic hyperbolic hierarchical module, thereby constructing a soft hierarchical relation matrix between the old and new classes in the hyperbolic space. Specifically, C t The construction process consists of the following four steps:

[0023] (a) The hyperbolic features are obtained through mapping. Given the superiority of hyperbolic space over Euclidean space in representing hierarchical structures, the exponential mapping operation of formula (1) is used. Projecting all Euclidean features z from Euclidean space onto hyperbolic space yields the corresponding hyperbolic features z. hyp :

[0024]

[0025] Where x0 is the origin of the mapping, and β is the absolute curvature of the Poincaré sphere. It is a conformal factor. It's the Möbius method.

[0026] (b) Calculate the hyperbolic class prototype for the new class. For each new class, calculate the hyperbolic mean Hmean operation using Equation (2). Based on its hyperbolic characteristics z hyp Calculate the prototype of the hyperbolic class

[0027]

[0028] P2K(x)=2x / (1+β|||x|| 2 ),

[0029]

[0030] Where P2K and K2P represent the coordinate transformation functions from the Poincaré sphere model to the Klein model and from the Klein model to the Poincaré sphere model, respectively, and γ x It is the Lorentz factor.

[0031] (c) Estimate the hierarchical relationship matrix. This is done using the hyperbolic distance metric operation in Equation (3). For old class and new categories Calculate the similarity as the hierarchical relationship matrix C. t The elements in [C] t ] i，j :

[0032]

[0033] Among them, z hyp , The results are given by formulas (1) and (2) respectively. Here, for the new class j, its hyperbolic prototype feature is directly taken. For the old class i, the hyperbolic feature is taken from all its hyperbolic characteristics. The most similar features.

[0034] (d) Normalization guided by hard ties. Finally, for C... tA hard-relationship-guided normalization strategy was designed to model clearer inter-class hierarchical relationships based on numerical differences between elements in the matrix. A shared knowledge matrix between new and old classes is defined.

[0035] ,

[0036] Where 1(·) is the indicator function, and σ is the sigmoid activation function. And K is thresholded. t To establish hard relationships between new and old classes that share a sufficient amount of knowledge, the following hard relationship matrix is ​​obtained.

[0037] [H t ] i,j =1([K t ] i,j ≥τ), (5)

[0038] Where τ = 0.6 is the threshold for hard relation determination. Therefore, based on H... t For the hierarchical relationship matrix C t Perform selective minimum-maximum normalization:

[0039]

[0040] Specifically, in the mission During the training process, there will be a certain number of training epochs at intervals. After each knowledge forward transfer, the above steps (a)-(d) are repeated to construct C. t The process improves the accuracy of hierarchical relationships through dynamic updates.

[0041] 3) Bidirectional knowledge transfer is achieved through the forward and backward transmission modules;

[0042] Obtain the current hierarchical relationship matrix C t Then, it is used as a transmission cost to guide the forward transmission module and the backward transmission module to carry out bidirectional knowledge transmission.

[0043] The purpose of the forward transfer module is to facilitate the learning of new classes, therefore it only acts on... The first half of the training epochs. Specifically, it involves the optimal transfer process of the new model using the old classifier, as shown in equation (7-10). The transmission generates an auxiliary new classifier.

[0044]

[0045] Where 1 is an all-one vector, the length of which is determined by the context, F tThe forward transport mapping matrix is ​​obtained by optimizing an optimal transport Kantorovitch form independent of model parameter updates using the Sinkhorn algorithm (Equation (8)); C t Let μ1 be the transmission cost matrix; the constraint terms μ1(μ2) are normalized marginal probabilities, representing the resource holdings of the transmission source in optimal transmission. Using the shared knowledge matrix of formula (4), the following class knowledge capacity strategy is designed to calculate the μ1 value representing the old class:

[0046]

[0047] Since the new class has not been fully learned during the forward propagation, the value of μ2 representing the new class is set to a uniform distribution:

[0048]

[0049] The purpose of the backward transfer module is to help with the memory of older classes, therefore it operates on... The second half of the training epochs. Similar to forward pass, it follows the optimal pass process according to equations (11-14), with the new class classifier of the new model passing through it. The output of the new class The transfer generated an auxiliary old class.

[0050]

[0051] Among them B t This is the backward transfer mapping matrix. Since both the old and new classes have been sufficiently learned, unlike forward transfer, the values ​​of the marginal probability constraints v1 and v2 are calculated using a class knowledge capacity strategy:

[0052]

[0053] 4) Construct the forward transmission loss and backward transmission loss;

[0054] Auxiliary new class classifier obtained from the forward transmission module It will be used to construct the forward propagation loss as shown in Equation (15). To leverage existing classes to facilitate the learning of new classes:

[0055]

[0056] in The loss is the binary cross-entropy loss, where dim(y) is the vector dimension.

[0057] Auxiliary old class obtained by the backward transmission module First, compare the logity output by the old model. t-1The weighted average was then used to construct the backpropagation loss as shown in Equation (16).

[0058]

[0059] Compared to distillation only y t-1 Traditional knowledge of distillation loss, The results of incorporating knowledge back-transmission This allows for more effective memorization of old knowledge by leveraging new categories.

[0060] 5) Optimize the total loss and update the model parameters;

[0061] In addition to acting on the first and second halves of the epochs respectively. and It also employs classification cross-entropy loss that applies to the entire process. To achieve basic supervised learning:

[0062]

[0063] in Let x be the one-heat truth label vector corresponding to the image sample x, and let x be the conventional distillation loss applied to the first half of the epochs. To achieve anti-forgetting during periods when backtransmission does not occur:

[0064]

[0065] In summary, the total loss for optimizing model parameters It is given by the following formula:

[0066]

[0067] Where λ=C t-1 / C t To balance the product factor of the number of new and old classes; α is the adjustment factor. Compared to The hyperparameter of intensity, let α = c t / C t-1 This reduces the amount of data as the number of tasks increases, mitigating the adverse effects of inaccurate forward transmission caused by increased forgetting in later stages.

[0068] For the above training environment setup, the program was run on a single NVIDIA RTX 3090 GPU using the PyTorch architecture. For parameter settings, the training epochs for each task on the IIRC-CIFAR dataset were 140, the batch size was 128, ResNet32 was used as the feature extractor, Stochastic Gradient Descent (SGD) was used as the optimizer, the learning rate was initialized to 1, the learning rate was adjusted to 0.1 of the current value during plateau periods, the number of replay samples saved per class was 20, the curvature β of the Poincaré sphere was 0.05, and the hierarchical relation matrix update period was 10 epochs; the training epochs for each task on the IIRC-ImageNet-Subset were as follows: The number of epochs is 100 (200 for the initial task), the learning rate is initialized to 0.5, the curvature β of the Poincaré sphere is 0.01, the period for updating the hierarchical relation matrix is ​​5 epochs, and the other parameter values ​​are the same as those of IIRC-CIFAR; IIRC-ImageNet-Lite uses ResNet50 as the feature extractor, and the other parameter values ​​are the same as those of IIRC-ImageNet-Subset; the parameter values ​​of IIRC-ImageNet-Full are the same as those of IIRC-ImageNet-Lite.

[0069] Furthermore, step 3 includes the following:

[0070] make This represents the test set corresponding to each task. These represent test image samples. In the mission The set of truth labels and the model The resulting set of predicted labels.

[0071] In the mission After training, the learned model will be used. For all seen tasks test set Prediction is performed under the condition of complete information, that is, the model needs to predict all known granular labels of the image sample, and calculate the index based on the prediction results: Average Precision Weighted Jaccard Similarity (APW-JS). After training on all T tasks, calculate the APW-JS metrics for all tasks. Taking the average, we obtain another Jaccard similarity (AIPW-JS) that measures the overall performance of all models by the average incremental precision weighted average. The specific calculation formulas for the two indicators are as follows:

[0072] Attached Figure Description

[0073] Figure 1 This is the task definition diagram for Incremental Implicit Thinning Image Classification (IIRC).

[0074] Figure 2 This is a technical flowchart of the present invention.

[0075] Figure 3 This is a schematic diagram of hyperbolic hierarchical modeling and bidirectional knowledge transfer network.

[0076] Figure 4 The curves show the changes in APW-JS Rj(↑) results on IIRC-CIFAR, IIRC-ImageNet-Subset, IIRC-ImageNet-Lite, and IIRC-ImageNet-Full.

[0077] Figure 5 It is IIRC-CIFAR Data set through model Θ 12 Visualization results of t-SNE features.

[0078] Figure 6 It is IIRC-CIFAR Data set through model Θ 15 Visualization results of t-SNE features. Detailed Implementation

[0079] This invention proposes an incremental implicit refinement image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transfer. The key idea of ​​this method is to continuously construct dynamic hierarchical relationships in hyperbolic space and use them as a transfer cost to guide the optimal bidirectional transfer of knowledge between old and new classes. This process establishes a feedback loop: the refined hierarchical relationships guide knowledge transfer, while the transfer benefits in turn affect the refinement of the hierarchical relationships. Hyperbolic hierarchical modeling is implemented through a dynamic hyperbolic hierarchical module, which fully utilizes the expressive power of hyperbolic geometry for hierarchy, embedding category features into the hyperbolic space to measure similarity. The resulting hyperbolic hierarchical relationships are dynamically updated after each knowledge transfer, thus benefiting from knowledge interaction. Bidirectional knowledge transfer is jointly implemented through a forward transfer module and a backward transfer module. Both modules use the current hierarchical relationship as the transfer cost. The forward transfer module transfers knowledge from the old class classifier to generate an auxiliary new class classifier, allowing the new class to inherit and effectively utilize the old knowledge. The backward transfer module transfers knowledge from the new class logit (the output of the new class classifier) ​​to generate an auxiliary old class logit, allowing the memorization of old knowledge to benefit from the learning of the new class. Furthermore, this invention proposes forward and backward transmission losses to further guide the two transmission processes, maximizing the effectiveness of each function. The main technical problems addressed by this method include the unsatisfactory performance of static hard-hierarchical relationships and insufficient knowledge utilization in current methods, thereby improving the accuracy of incremental implicit refinement image classification.

[0080] As shown in the description Figure 2 As shown, the technical solution of the present invention is implemented according to the following steps:

[0081] Step 1, Incremental task partitioning of the dataset: Reconstruct the original image dataset into a dataset with a two-level hierarchical structure, and partition the dataset for each task.

[0082] Step 2, Training the Network: Input the training set of the current task, train the hyperbolic hierarchical modeling and bidirectional knowledge transfer network, and update the model parameters;

[0083] Step 3, Test and output classification accuracy: Test on the test set of all known tasks and output an average incremental classification accuracy;

[0084] Repeat steps 2 and 3 until no new tasks appear.

[0085] The specific instructions for each step are as follows:

[0086] 1. Incrementally divide the dataset into tasks.

[0087] To simulate the scenario of incremental implicit refinement in image classification, we first reconstruct the original image classification dataset (which contains no hierarchical relationship between categories, i.e., only subclasses) into a two-level hierarchical dataset containing parent and child classes. Then, according to a certain task division setting, we distribute each category into each task in the incremental task sequence to obtain the final training and test sets for each task.

[0088] This invention employs four datasets with a two-level hierarchical structure: IIRC-CIFAR, IIRC-ImageNet-Full, IIRC-ImageNet-Lite, and IIRC-ImageNet-Subset. These datasets are reconstructed from two commonly used open-source image classification datasets, CIFAR100 and ImageNet, with the latter two being lightweight versions of IIRC-ImageNet-Full. For each hierarchical dataset, the parent class's data is sampled from each of its corresponding child class data and then merged. Each child class's data is composed of 80% of its original data, meaning that the parent class and each of its child classes share 20% of the data. Specifically, the IIRC-CIFAR dataset has 15 parent classes and 100 child classes, with each parent class corresponding to 4-8 child classes. It is divided into 22 tasks, with the initial task containing 10 parent classes and each subsequent task containing 5 classes. The IIRC-ImageNet-Full dataset has 85 parent classes and 998 child classes, with each parent class corresponding to 3-118 child classes. It is divided into 35 tasks, with the initial task containing 63 parent classes and each subsequent task containing 30 classes. The IIRC-ImageNet-Lite dataset has 69 parent classes and 264 child classes, and it is divided into 10 tasks, with the initial task containing 63 parent classes and each subsequent task containing 30 classes. The IIRC-ImageNet-Subset dataset has 10 parent classes and 100 child classes, and it is divided into 11 tasks. Unlike the previous three datasets, each task in this dataset has 10 classes, and the 10 parent classes are evenly distributed across the first ten tasks.

[0089] 2. Training the network

[0090] For each task, we input the training set of the current task, train the established hyperbolic hierarchical modeling and bidirectional knowledge transfer network, and update the model parameters.

[0091] As shown in the description Figure 3 As shown, the training process can be divided into the following 5 steps:

[0092] 6) Input image data to obtain model output

[0093] Given a task sequence The t-th task The training set is defined as in These consist of an image set and a label set (labels for different tasks do not overlap, i.e.) These are the i-th input image sample and its corresponding single label, The number of samples in the training set; the replay set is defined as Its old task Sampled from the training data, and compared with the training set Participate in the current mission together Learning to resist catastrophic forgetting. For ease of narration, record the current task. The number of category labels is remember The total number of category tags is Training task At that time, firstly, each image data Input the old model Θ respectively t-1 =(Φ t-1 Ψ t-1 ) and the new model Θ t =(Φ t Ψ t This will allow you to obtain some of the necessary output results, including... For feature extractors, For classifier (Ψ) t It can be divided into two parts: the old class classifier. And new class classifier On the one hand, x is input to the old model Θ from the previous task, whose parameters are now frozen. t-1 Obtain the old model classifier Ψ t-1 Output: Old Model On the other hand, x is the input to the new model Θ for the current task. t Obtain the new model feature extractor Φ t Output: Euclidean features The old classifier for the new model Output: Old Class and new class classifiers Output: New class

[0094] 7) Construct a hierarchical relationship matrix using a dynamic hyperbolic hierarchical module.

[0095] Next, we will obtain the training set and replay set The Euclidean features z of all samples are input into the dynamic hyperbolic hierarchical module, thereby constructing a soft hierarchical relation matrix between the old and new classes in the hyperbolic space. Specifically, Ct The construction process consists of the following four steps:

[0096] (a) The hyperbolic features are obtained through mapping. Given the superiority of hyperbolic space over Euclidean space in representing hierarchical structures, the exponential mapping operation of formula (1) is used. Projecting all Euclidean features z from Euclidean space onto hyperbolic space (Poincaré sphere) yields the corresponding hyperbolic features z. hyp :

[0097]

[0098] Where x0 is the origin of the mapping (generally set, x0 = 0), and β is the absolute curvature of the Poincaré sphere. It is a conformal factor. It's the Möbius method.

[0099] (b) Calculate the hyperbolic class prototype for the new class. For each new class, calculate the hyperbolic mean Hmean operation using Equation (2). Based on its hyperbolic characteristics z hyp Calculate the prototype of the hyperbolic class

[0100]

[0101] P2K(x)=2x / (1+β||x|| 2 ),

[0102]

[0103] Where P2K and K2P represent the coordinate transformation functions from the Poincaré sphere model to the Klein model and from the Klein model to the Poincaré sphere model, respectively, and γ x It is the Lorentz factor.

[0104] (c) Estimate the hierarchical relationship matrix. We perform this using the hyperbolic distance metric operation in Equation (3). For old class and new categories Calculate the similarity as the hierarchical relationship matrix C. t The elements in [C] t ] i，j :

[0105]

[0106] Among them, z hyp , The results are given by formulas (1) and (2) respectively. Here, for the new class j, we directly take its hyperbolic prototype feature. For the old class i, we take the hyperbolic features from all its hyperbolic characteristics. The most similar feature, this operation avoids the limited old class replay samples. This addresses the unreliability of class prototype representations, but it also allows for the full utilization of shared instances between parent and child classes to help estimate more accurate hierarchical relationships.

[0107] (d) Normalization guided by hard ties. Finally, we normalize C. t We designed a hard-relationship-guided normalization strategy to model clearer inter-class hierarchical relationships based on numerical differences between elements in the matrix. Specifically, we defined a shared knowledge matrix between new and old classes.

[0108] ,

[0109] Where 1(·) is the indicator function, and σ is the sigmoid activation function. And K is thresholded. t To establish hard relationships between new and old classes that share a sufficient amount of knowledge, the following hard relationship matrix is ​​obtained.

[0110] [H t ] i,j =1([K t ] i,j ≥τ),(25)

[0111] Where τ = 0.6 is the threshold for hard relation determination. Therefore, based on H... t For the hierarchical relationship matrix C t Perform selective minimum-maximum normalization:

[0112]

[0113] Specifically, in the mission During the training process, we will periodically perform a certain number of training epochs (controlled by the hyperparameter period). After each forward (backward) knowledge transfer, we will repeat steps (a)-(d) above to construct C. t The process improves the accuracy of hierarchical relationships through dynamic updates.

[0114] 8) Bidirectional knowledge transfer is achieved through the forward and backward transmission modules.

[0115] Obtain the current hierarchical relationship matrix C t Then, it is used as a transmission cost to guide the forward transmission module and the backward transmission module to carry out bidirectional knowledge transmission.

[0116] The purpose of the forward transfer module is to facilitate the learning of new classes, therefore it only acts on... The first half of the training epochs. Specifically, it involves the optimal transfer process of the new model using the old classifier, as shown in equation (7-10). The transmission generated an auxiliary new classifier.

[0117]

[0118] Where 1 is an all-one vector, the length of which is determined by the context, F t The forward transport mapping matrix is ​​obtained by optimizing an optimal transport Kantorovitch form independent of model parameter updates using the Sinkhorn algorithm (Equation (8)); C t The transmission cost matrix (i.e., the hierarchical cost matrix) is used; the constraint terms μ1 (μ2) are normalized marginal probabilities, which represent the resource holding (demand) of the transmission source (target) in optimal transmission. In this invention, they actually reflect the ability of an old (new) class to transmit knowledge to all new (old) classes. Through experiments, we found that the contribution of each old class to the new class varies depending on the amount of knowledge it holds. Therefore, we make full use of the shared knowledge matrix in formula (4) and design the following class knowledge capacity strategy to calculate the μ1 value representing the old class:

[0119]

[0120] Since the new class has not been fully learned during the forward propagation, the value of μ2 representing the new class is set to a uniform distribution:

[0121]

[0122] The purpose of the backward transfer module is to help with the memory of older classes, therefore it operates on... The second half of the training epochs. Similar to forward pass, it follows the optimal pass process according to equations (11-14), with the new class classifier of the new model passing through it. The output of the new class The transfer generated an auxiliary old class.

[0123]

[0124]

[0125] Among them B t This is the backward transfer mapping matrix. Since both the old and new classes have been sufficiently learned, unlike forward transfer, the values ​​of the marginal probability constraints v1 and v2 are calculated using a class knowledge capacity strategy:

[0126]

[0127] 9) Construct the forward transmission loss and backward transmission loss

[0128] Auxiliary new class classifier obtained from the forward transmission module It will be used to construct the forward propagation loss as shown in Equation (15). To leverage existing classes to facilitate the learning of new classes:

[0129]

[0130] in The loss is the binary cross-entropy loss, where dim(y) is the vector dimension.

[0131] Auxiliary old class obtained by the backward transmission module First, compare the logity output by the old model. t-1 The weighted average was then used to construct the backpropagation loss as shown in Equation (16).

[0132]

[0133] Compared to distillation only y t-1 Traditional knowledge of distillation loss, The results of incorporating knowledge back-transmission This allows for more effective memorization of old knowledge by leveraging new categories.

[0134] 10) Optimize total loss and update model parameters

[0135] In addition to acting on the first and second halves of the epochs respectively. and It also employs classification cross-entropy loss that applies to the entire process. To achieve basic supervised learning:

[0136]

[0137] in Let x be the one-hot ground truth label vector corresponding to the image sample x (since the training condition is incomplete, ...). (In the form of single-heat distillation). And the traditional distillation losses that occur in the first half of the epochs. To achieve anti-forgetting during periods when backtransmission does not occur:

[0138]

[0139] In summary, the total loss for optimizing model parameters It can be given by the following formula:

[0140]

[0141] Where λ=C t-1 / C t To balance the product factors of the number of new and old classes; α is for adjustment. Compared to The hyperparameter of intensity, let α = c t / C t-1 This reduces the amount of data as the number of tasks increases, mitigating the adverse effects of inaccurate forward transmission caused by increased forgetting in later stages.

[0142] For the training environment settings described above, this invention uses the PyTorch architecture to run the program on a single NVIDIA RTX3090 GPU. For parameter settings, the training epochs for each task on the IIRC-CIFAR dataset are 140 (initially 280 for the initial task), the batch size is 128, ResNet32 is used as the feature extractor, stochastic gradient descent (SGD) is used as the optimizer, the learning rate is initialized to 1, the learning rate is adjusted to 0.1 of the current value during plateau periods, the number of replay samples saved per class is 20, the curvature β of the Poincaré sphere is 0.05, and the period for updating the hierarchical relation matrix is ​​10 epochs; for each task on the IIRC-ImageNet-Subset... The training epochs for the task are 100 (the initial task is 200), the learning rate is initialized to 0.5, the curvature β of the Poincaré sphere is 0.01, and the update period of the hierarchical relation matrix is ​​5 epochs. The other parameter values ​​are the same as those for IIRC-CIFAR. IIRC-ImageNet-Lite uses ResNet50 as the feature extractor, and the other parameter values ​​are the same as those for IIRC-ImageNet-Subset. The parameter values ​​for IIRC-ImageNet-Full are the same as those for IIRC-ImageNet-Lite.

[0143] 3. Test and output classification accuracy.

[0144] make This represents the test set corresponding to each task. These represent test image samples. In the mission The set of truth labels and the model The resulting set of predicted labels.

[0145] In the mission After training, we will use the learned model For all seen tasks test set Prediction is performed under the condition of complete information, that is, the model needs to predict all known granular labels of the image sample, and calculate the index based on the prediction results: Average Precision Weighted Jaccard Similarity (APW-JS). After training on all T tasks, we will calculate the APW-JS metrics for all tasks. Taking the average, we obtain another Jaccard similarity (AIPW-JS) that measures the overall performance of all models by the average incremental precision weighted average. The specific calculation formulas for the two indicators are as follows:

[0146]

[0147] This invention conducts quantitative and qualitative experimental verification of the effectiveness of the proposed technology. The quantitative evaluation provides a comparison of the classification accuracy of this technology with existing technologies on four datasets, including IIRC-CIFAR, proving the effectiveness of this technology in solving the incremental implicit refinement image classification problem. The qualitative evaluation provides the t-SNE feature visualization results of this technology on the IIRC-CIFAR dataset, illustrating the role of the key feature of this technology—hierarchical guided bidirectional knowledge transfer.

[0148] Quantitative assessment

[0149] Table 1 lists the classification accuracy metrics of all techniques on the IIRC-CIFAR, IIRC-ImageNet-Subset, and IIRC-ImageNet-Lite datasets using the AIPW-JS benchmark. RTDHH is an alternative name for this technique. Bold and underlined text represent the best and second-best results, respectively. "w / IfH" indicates that the Infer-HCV strategy proposed in the existing HCV technique is used to correct predictions during inference using learned hard relations. We list the results under the "w / IfH" and "w / o IfH" settings separately for a fairer comparison. The table shows that under both settings, RTDHH demonstrates superior performance on all datasets, outperforming all methods except the first three, which benefit from access to complete data or labels. Specifically, on the three datasets, RTDHH improves performance by 0.0153, 0.0321, and 0.0286 compared to the second-best methods under "w / o IfH," and by 0.0076 and 0.0025 under "w / IfH." Table 1: AIPW-JS R (↑) on IIRC-CIFAR, IIRC-ImageNet-Subset, and IIRC-ImageNet-Lite

[0150] result

[0151] Figure 3 This shows the results of all techniques on the APW-JS metric across four datasets, including IIRC-ImageNet-Full.

[0152]

[0153] The IIRC-ImageNet-Full dataset is expensive to train, and only a few samples were computed. (Results on the above). Similarly, RTDHH achieved state-of-the-art performance under different settings. On the more challenging ImageNet dataset, RTDHH without Infer-HCV-assisted inference even outperformed the previous state-of-the-art method HCV using Infer-HCV, fully demonstrating the effectiveness of the proposed technique RTDHH.

[0154] Qualitative assessment

[0155] In order to analyze the function of the forward transmission module, Figure 5 This study compared our technique (RTDHH) with the baseline technique (iCaRL_CNN) in IIRC-CIFAR. Dataset based on model Θ 12 The resulting t-SNE feature visualizations show that sibling classes belong to the same parent class. It can be observed that the baseline technique iCaRL_CNN exhibits close grouping between different groups, with significant overlap between sibling classes within the same group. In contrast, our proposed RTDH technique enhances intra-group compactness through a forward pass module, effectively increasing inter-group distance. Furthermore, under the constraint of the classification loss, RTDH learns more discriminative features for sibling classes, reducing overlap between them.

[0156] In order to analyze the function of the backward transmission module, Figure 6 The results of comparing RTDHH and iCaRL_CNN in IIRC-CIFAR were then analyzed. Dataset based on model Θ 15 The resulting t-SNE feature visualization results show that the baseline technique iCaRL_CNN, relying solely on the traditional method of knowledge distillation from frozen old models, cannot effectively resist the catastrophic forgetting of hierarchical relationships. The visualization results show the drift of subclass distributions relative to their parent class distributions. In contrast, RTDHH, through the backpropagation module, fully utilizes the knowledge of new classes to strengthen the hierarchical relationships, achieving a comprehensive wrapping of the parent class distribution with its subclass distributions.

Claims

1. An incremental implicit thinning image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transfer, characterized in that, Includes the following steps: Step 1, Incremental task partitioning of the dataset: Reconstruct the original image dataset into a dataset with a two-level hierarchical structure, and partition the dataset for each task. Step 2, Training the Network: Input the training set of the current task, train the hyperbolic hierarchical modeling and bidirectional knowledge transfer network, and update the model parameters; Step 3, Test and output classification accuracy: Test on the test set of all known tasks and output an average incremental classification accuracy; Repeat steps 2 and 3 until no new tasks appear.

2. The incremental implicit thinning image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transfer according to claim 1, characterized in that, Step 1, incremental task partitioning of the dataset, specifically includes the following: To simulate the scenario of incremental implicit refinement of image classification, the original image classification dataset is first reconstructed into a two-level hierarchical dataset containing parent and child classes. Then, according to a certain task division setting, each category is distributed in each task of the incremental task sequence to obtain the final training and test sets for each task. Four datasets with two-level hierarchical structures were used: IIRC-CIFAR, IIRC-ImageNet-Full, IIRC-ImageNet-Lite, and IIRC-ImageNet-Subset. These datasets were reconstructed from two commonly used open-source image classification datasets, CIFAR100 and ImageNet, with the latter two being lightweight versions of IIRC-ImageNet-Full. For each hierarchical dataset, the data for each parent class was sampled from each of its corresponding child classes and then merged. The data for each child class consisted of 80% of its original data, meaning that the parent class and each of its child classes shared 20% of the data. Specifically, the IIRC-CIFAR dataset had 15 parent classes and 100 child classes, with each parent class corresponding to 4–8 child classes. The first dataset is divided into 22 tasks, with the initial task containing 10 parent classes and each subsequent task containing 5 classes. The IIRC-ImageNet-Full dataset has 85 parent classes and 998 child classes, with each parent class corresponding to 3 to 118 child classes. It is divided into 35 tasks, with the initial task containing 63 parent classes and each subsequent task containing 30 classes. The IIRC-ImageNet-Lite dataset has 69 parent classes and 264 child classes. It is divided into 10 tasks, with the initial task containing 63 parent classes and each subsequent task containing 30 classes. The IIRC-ImageNet-Subset dataset has 10 parent classes and 100 child classes. It is divided into 11 tasks. Unlike the first three datasets, each task has 10 classes, and the 10 parent classes are evenly distributed across the first ten tasks.

3. The incremental implicit thinning image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transfer according to claim 1, characterized in that, The training network in step 2 includes the following: For each task, the training set of the current task is input to train the established hyperbolic hierarchical modeling and bidirectional knowledge transfer network, and the model parameters are updated. The training process consists of the following 5 steps: 1) Input image data to obtain model output; Given a task sequence The t-th task The training set is defined as in These are the image set and the label set, respectively. The labels for different tasks do not overlap. These are the i-th input image sample and its corresponding single label, The number of samples in the training set; the replay set is defined as Its old task Sampled from the training data, and compared with the training set Participate in the current mission together Learning; recording current tasks The number of category labels is remember The total number of category tags is Training task At that time, firstly, each image data Input the old model Θ respectively t-1 =(Φ t-1 Ψ t-1 ) and the new model Θ t =(Φ t Ψ t This will allow you to obtain some of the necessary output results, including... For feature extractors, For classifiers, Ψ t It can be divided into two parts: the old class classifier. And new class classifier On the one hand, x is input to the old model Θ from the previous task, whose parameters are now frozen. t-1 Obtain the old model classifier Ψ t-1 Output of the old model logit On the other hand, x is the input to the new model Θ for the current task. t Obtain the new model feature extractor Φ t Output: Euclidean features The old classifier for the new model Output: Old Class and new class classifiers Output: New class 2) Construct a hierarchical relationship matrix using a dynamic hyperbolic hierarchical module; The obtained training set and replay set The Euclidean features z of all samples are input into the dynamic hyperbolic hierarchical module to construct a soft hierarchical relation matrix between the old and new classes in hyperbolic space. 3) Bidirectional knowledge transfer is achieved through the forward and backward transmission modules; Obtain the current hierarchical relationship matrix C t Then, it is used as a transmission cost to guide the forward transmission module and the backward transmission module to carry out bidirectional knowledge transmission; The purpose of the forward transfer module is to facilitate the learning of new classes, therefore it only acts on... The first half of the training epochs; specifically, it involves the optimal transfer process of the new model using the old classifier through formula (7-10). The transmission generates an auxiliary new classifier. Where 1 is an all-one vector, the length of which is determined by the context, F t The forward transport mapping matrix is ​​obtained by optimizing an optimal transport Kantorovitch form (8) independent of model parameter updates using the Sinkhorn algorithm; C t Let μ1 be the transmission cost matrix; the constraint terms μ1(μ2) are normalized marginal probabilities, representing the resource holdings of the transmission source in optimal transmission. Using the shared knowledge matrix of formula (4), the following class knowledge capacity strategy is designed to calculate the μ1 value representing the old class: Since the new class has not been fully learned during the forward propagation, the value of μ2 representing the new class is set to a uniform distribution: The purpose of the backward transfer module is to help with the memory of older classes, and it functions as follows: The second half of the training epochs; similar to forward propagation, it is carried out through the optimal propagation process of formula (11-14), by the new class classifier of the new model. The output of the new class The transfer generated an auxiliary old class. Among them B t This is the backward transfer mapping matrix; since both the old and new classes have been fully learned, unlike forward transfer, the values ​​of the marginal probability constraints v1 and v2 are calculated using the class knowledge capacity strategy: 4) Construct the forward transmission loss and backward transmission loss; Auxiliary new class classifier obtained from the forward transmission module It will be used to construct the forward propagation loss as shown in Equation (15). To leverage existing classes to facilitate the learning of new classes: Among them l BCE (·,·) represents the binary cross-entropy loss, and dim(y) represents the vector dimension; Auxiliary old class obtained by the backward transmission module First, compare the logity output by the old model. t-1 The weighted average was then used to construct the backpropagation loss as shown in Equation (16). Compared to distillation only y t-1 Traditional knowledge of distillation loss, The results of incorporating knowledge back-transmission This allows for more effective memorization of old knowledge by leveraging new categories; 5) Optimize the total loss and update the model parameters; In addition to acting on the first and second halves of the epochs respectively. and Employing classification cross-entropy loss that applies throughout the entire process To achieve basic supervised learning: in The vector of uniquely heated true values ​​for image sample x; and the conventional distillation loss applied to the first half of the epochs. To achieve anti-forgetting during periods when backtransmission does not occur: In summary, the total loss for optimizing model parameters It is given by the following formula: Where λ=C t-1 / C t To balance the product factor of the number of new and old classes; α is the adjustment factor. Compared to The hyperparameter of intensity, let α = c t / C t-1 This reduces the amount of data as the number of tasks increases, mitigating the adverse effects of inaccurate forward transmission caused by increased forgetting in later stages.

4. The incremental implicit thinning image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transfer according to claim 3, characterized in that, For the above training environment setup, the program was run on a single NVIDIA RTX3090 GPU using the PyTorch architecture. For parameter settings, the training epochs for each task on the IIRC-CIFAR dataset were 140, the batch size was 128, ResNet32 was used as the feature extractor, stochastic gradient descent (SGD) was used as the optimizer, the learning rate was initialized to 1, the learning rate was adjusted to 0.1 of the current value during plateau periods, the number of replay samples saved per class was 20, the curvature β of the Poincaré sphere was 0.05, and the hierarchical relation matrix update period was 10 epochs. IIRC-ImageNet-S The training epochs for each task on ubset are 100, the learning rate is initialized to 0.5, the curvature β of the Poincaré sphere is 0.01, and the period for updating the hierarchical relation matrix is ​​5 epochs. The other parameter values ​​are the same as those of IIRC-CIFAR. IIRC-ImageNet-Lite uses ResNet50 as the feature extractor, and the other parameter values ​​are the same as those of IIRC-ImageNet-Subset. The parameter values ​​of IIRC-ImageNet-Full are the same as those of IIRC-ImageNet-Lite.

5. The incremental implicit thinning image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transfer according to claim 3, characterized in that, Specifically, C t The construction process consists of the following four steps: (a) The hyperbolic feature is obtained by mapping; given the superiority of hyperbolic space over Euclidean space in hierarchical structure representation, the exponential mapping operation of formula (1) is used. Projecting all Euclidean features z from Euclidean space onto hyperbolic space yields the corresponding hyperbolic features z. hyp : Where x0 is the origin of the mapping, and β is the absolute curvature of the Poincaré sphere. It is a conformal factor. It is the Möbius method; (b) Calculate the hyperbolic class prototype of the new class; for each new class, calculate the hyperbolic mean Hmean operation using formula (2). Based on its hyperbolic characteristics z hyp Calculate the prototype of the hyperbolic class Where P2K and K2P represent the coordinate transformation functions from the Poincaré sphere model to the Klein model and from the Klein model to the Poincaré sphere model, respectively, and γ x Lorentz factor; (c) Estimate the hierarchical relationship matrix; using the hyperbolic distance metric operation of Equation (3). For old class and new categories Calculate the similarity as the hierarchical relationship matrix C. t The elements in [C] t ] i，j : in, The results are given by formulas (1) and (2) respectively; here, for the new class j, its hyperbolic prototype feature is directly taken. For the old class i, the hyperbolic feature is taken from all its hyperbolic characteristics. Most similar features; (d) Normalization guided by hard ties; finally, for C t A hard-relationship-guided normalization strategy was designed to enable the numerical differences between elements in the matrix to model clearer inter-class hierarchical relationships; a shared knowledge matrix between new and old classes was defined. Where 1(·) is the indicator function, σ is the sigmoid activation function, and K is thresholded. t To establish hard relationships between new and old classes that share a sufficient amount of knowledge, the following hard relationship matrix is ​​obtained. [H t ] i,j =1([K t ] i,j ≥τ), (18) Where τ = 0.6 is the threshold for hard relation determination; therefore, according to H t For the hierarchical relationship matrix C t Perform selective minimum-maximum normalization: Specifically, in the mission During the training process, there will be a certain number of training epochs at intervals. After each knowledge forward transfer, the above steps (a)-(d) are repeated to construct C. t The process improves the accuracy of hierarchical relationships through dynamic updates.

6. The incremental implicit thinning image classification method based on hyperbolic hierarchical modeling and bidirectional knowledge transfer according to claim 1, characterized in that, Step 3 includes the following: make This represents the test set corresponding to each task. These represent test image samples. In the mission The set of truth labels and the model The resulting set of predicted labels; In the mission After training, the learned model will be used. For all seen tasks test set Prediction is performed under the condition of complete information, that is, the model needs to predict all known granular labels of the image sample, and calculate the index based on the prediction results: Average Precision Weighted Jaccard Similarity (APW-JS). After training on all T tasks, calculate the APW-JS metrics for all tasks. Taking the average, we obtain another Jaccard similarity (AIPW-JS) that measures the overall performance of all models by the average incremental precision weighted average. The specific calculation formulas for the two indicators are as follows: 。

Images