A method and apparatus for compressing and inheriting ancestor model knowledge to descendant models through optimal transmission

Abstract

This invention discloses a method and apparatus for compressing and inheriting ancestor model knowledge into descendant models through optimal transmission. The method includes: constructing an ancestor model containing multiple Transformer network layers and obtaining a unified feature representation for each layer of the ancestor model; constructing a difference metric between each layer of the ancestor model and candidate learning genes; modeling the cross-layer knowledge mapping process as an entropy-regularized optimal transmission problem; solving the entropy-regularized optimal transmission problem to obtain a cross-layer knowledge transmission matrix; performing cross-layer weighted fusion of ancestor model parameters based on the cross-layer knowledge transmission matrix to construct shared learning gene parameters; constructing descendant models of different scales based on the learning gene parameters; and applying the constructed descendant models to the training and inference of downstream tasks. This invention can complete structured knowledge inheritance and model reconstruction without accessing the original training data of the ancestor model, and has good deployability and generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a method and apparatus for compressing ancestor model knowledge and inheriting it to descendant models through optimal transmission. Background Technology

[0002] With the development of deep learning, the Vision Transformer (ViT) has made significant progress in tasks such as image recognition, object detection, and multimodal learning. As Vision Transformer (ViT) is widely applied in various vision tasks, the complex representation capabilities inherent in its deep structure have been confirmed by numerous studies. Compared to convolutional neural networks, ViT models global dependencies through a self-attention mechanism, enabling it to extract visual features more effectively. However, with the continuous increase in model depth and width, the parameter scale expands rapidly, leading to a significant increase in training costs, storage overhead, and inference latency, making it difficult to deploy large-scale ViT models on resource-constrained devices or in real-time applications. Furthermore, the parameters between different layers exhibit high dimensionality, inconsistent distribution, and latent correlations that are difficult to observe directly, making it extremely difficult to uniformly model the internal knowledge of the model.

[0003] To address the aforementioned issues, existing technologies have proposed methods such as model compression, efficient parameter fine-tuning (PEFT), low-rank decomposition, and distillation to reduce model size or improve transfer efficiency. However, these methods still have significant limitations:

[0004] (1) Lack of a consistent knowledge extraction mechanism across layers. Most existing methods decompose or compress the parameters of each layer independently, failing to reveal the potential shared feature patterns between the multi-layer structure of the Transformer, and thus cannot construct a unified parameter space for cross-layer transfer. Existing technologies cannot obtain the structural information shared across layers, let alone understand the stable expression patterns formed by the model in each layer from a holistic perspective. Due to the lack of a systematic cross-layer analysis mechanism, a large amount of reusable structural knowledge that may exist inside the Transformer has not been fully explored, resulting in limited efficiency in model compression, transfer, and structural reconstruction.

[0005] (2) It is difficult to explain the structural commonalities between different layers. The multi-layer self-attention structure of ViT has highly similar computational forms, but existing studies lack analysis of the distribution patterns of parameters in different layers, lack effective methods to identify and extract such cross-layer commonalities, and cannot identify the reusable "core knowledge" that appears repeatedly in each layer.

[0006] (3) There is a lack of theoretical methods for model expansion or scaling. Currently, the construction of ViT models of different scales (such as Tiny, Small, Base, etc.) mainly relies on manual design of depth or width, and there is a lack of a unified mechanism to automatically generate descendant models using the internal knowledge of ancestor models.

[0007] (4) Transfer learning still relies on a large amount of downstream data and training costs. In existing methods, building lightweight models or adapting to new tasks requires a lot of fine-tuning training, and it is impossible to directly extract transferable structural knowledge from existing large models, resulting in slow transfer speed and unstable performance.

[0008] Therefore, how to automatically extract the core knowledge structure that is shared across layers, expressive and reusable from the fully trained ViT ancestor model, and then use this knowledge to build descendant models of different depths and parameter scales, is a key technical problem in current model compression, effective transfer and efficient deployment. Summary of the Invention

[0009] To address the problems existing in the prior art, this invention proposes a method and apparatus for compressing and inheriting ancestral model knowledge into descendant models through optimal transmission. By uniformly modeling, decomposing, and recombining the parameters of each layer of the ancestral model, "learning genes" capable of cross-task and cross-model scale transfer are extracted. These learning genes are then used to construct descendant models of different depths or capacities, thereby achieving model compression, rapid adaptation, and efficient training. Structured knowledge inheritance and model reconstruction can be completed without accessing the original training data of the ancestral model, exhibiting good deployability and generalization ability.

[0010] This invention constructs an ancestor model containing multiple network layers, performs alignment measurement on the intermediate features of each layer of the ancestor model, learns cross-layer knowledge transfer relationships through entropy regularization of the optimal transfer method, thereby extracting transferable learning genes; and constructs descendant models of different depths or widths based on the learning genes, enabling the descendant models to achieve fast convergence and accurate inference on unseen downstream tasks.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A method for compressing ancestor model knowledge and inheriting it to descendant models through optimal transmission includes the following steps:

[0013] Step S1: Construct an ancestor model containing multiple Transformer network layers and obtain a unified feature representation for each layer of the ancestor model;

[0014] Step S2: Construct a measure of the difference between each layer of the ancestor model and the candidate learning genes;

[0015] Step S3: Model the cross-layer knowledge mapping process as an entropy-regularized optimal transmission problem;

[0016] Step S4: Solve the entropy regularization optimal transmission problem to obtain the cross-layer knowledge transmission matrix;

[0017] Step S5: Perform cross-layer weighted fusion of ancestor model parameters based on the cross-layer knowledge transfer matrix to construct shared learning gene parameters;

[0018] Step S6: Construct offspring models of different scales based on the learned gene parameters;

[0019] Step S7: Apply the completed descendant model to the training and inference of downstream tasks.

[0020] Furthermore, step S1 specifically includes:

[0021] Construct an ancestor model containing multiple Transformer network layers, and input samples Forward propagation is performed on the ancestor model; in the ancestor model... In each layer, the category label embedding vector corresponding to that layer is extracted. And the attention map output by the multi-head self-attention mechanism ,in Indicates the number of heads of attention. Indicates the sequence length. Label the embedding vector for the category. The feature dimension of the category label embedding vector is represented. For attention maps;

[0022] Normalization is performed on the category label embedding vector and attention map respectively, and the normalized features are concatenated along the feature dimensions to obtain the first... The unified characteristics of a layer are represented as follows:

[0023] ,

[0024] Among the symbols This represents a vector-level concatenation operation. This is the normalization function.

[0025] Furthermore, step S2 specifically includes:

[0026] For the ancestor model Layer feature representation Feature representation of candidate learning gene layers Perform a difference measurement, where Initialize vectors for trainable, learnable gene features. Indicates the index number of the learning gene. The dimension of the learned gene feature vector is represented; the Euclidean squared distance is used as the difference measure, and the calculation formula is as follows:

[0027] ,

[0028] A cross-layer feature difference matrix is ​​constructed by calculating the differences between all layers of the ancestor model and the learned genes. .

[0029] Furthermore, step S3 specifically includes:

[0030] With cross-layer transmission matrix To optimize the variables, construct the objective function:

[0031] ,

[0032] in, Here is the entropy regularization coefficient. This represents the cost of cross-layer feature matching; simultaneously, a mass conservation constraint is applied.

[0033] ,

[0034] in This is a normalized hierarchical weight distribution vector to ensure that cross-layer transmission satisfies probability conservation.

[0035] Furthermore, step S4 employs an iterative scaling method to solve the entropy-regularized optimal transmission problem, specifically including:

[0036] First, construct the kernel matrix based on the cross-layer feature difference matrix and the entropy regularization coefficient:

[0037] ,

[0038] Then, the scaling vector is updated alternately:

[0039] ,

[0040] The final cross-layer knowledge transfer matrix is ​​obtained as follows:

[0041] ,

[0042] Among the symbols This indicates element-wise division;

[0043] After multiple iterations until convergence, a cross-layer knowledge transfer matrix satisfying the regularization condition and probability constraints is obtained. .

[0044] Furthermore, step S5 specifically includes:

[0045] When the ancestor model parameters are attention weight parameters, let the ancestor model be... The attention weights of the layer are Then the first Each learning gene parameter is represented as:

[0046] .

[0047] Furthermore, the construction of descendant models of different scales in step S6 includes any one of the following methods:

[0048] (1) Reuse the learned gene parameters in multiple network layers to construct a deep adjustable model structure;

[0049] (2) The learning gene parameters are transformed into neural network structure parameters of different dimensions. The calculation method is as follows:

[0050] ,

[0051] in, It is a linear mapping matrix.

[0052] Furthermore, in step S7, the training of the offspring model in the downstream task adopts a full parameter fine-tuning method, performs end-to-end optimization on all trainable parameters on the network structure initialized by learning genes, and uses stochastic gradient descent-type optimization methods to update all network parameters on the downstream dataset.

[0053] Furthermore, an image classification method includes:

[0054] Collect image data and form a dataset;

[0055] The dataset is input and a descendant model is constructed using a method that compresses and inherits the knowledge of the ancestor model through optimal transmission. This model is then used for training, inference, and outputting image classification results.

[0056] The present invention also provides an apparatus for compressing and inheriting ancestor model knowledge into a descendant model through optimal transmission, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method for compressing and inheriting ancestor model knowledge into a descendant model through optimal transmission.

[0057] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0058] This invention constructs a unified feature representation for multi-layer Transformers and introduces entropy regularization to learn cross-layer correspondences through optimal transport learning, enabling the systematic extraction of potential common structures across layers of the ancestor model. Compared to traditional methods that only compress or decompose within a single layer, this invention can capture consistent semantic patterns and structural regularities across layers, thereby significantly improving the completeness and stability of knowledge extraction.

[0059] This invention utilizes cross-layer weighted fusion to construct compact learning gene parameters, preserving the key structures of the ancestral model and enabling knowledge transfer without accessing the original training data. Compared to existing technologies that rely on large-scale data distillation or complex fine-tuning strategies, this invention exhibits lower data dependence and higher transfer efficiency.

[0060] This invention provides bidirectional scaling capabilities with adjustable depth and width during the model reconstruction stage. Through two mechanisms—weight sharing and weight concatenation—it can automatically generate multi-scale descendant models based on different computational budgets. Compared to existing model scaling methods that require manual structural design or extensive experimental adjustments, this invention offers greater flexibility and consistency.

[0061] Because the progeny model is initialized with the learning genes, it exhibits faster convergence and better generalization performance in downstream tasks, especially under conditions of few-sample training. Compared with random initialization or other learning methods, this invention can achieve higher accuracy in a shorter training time, significantly improving the adaptation efficiency of downstream tasks.

[0062] In summary, this invention automatically extracts a consistent, expressive, and reusable core knowledge structure from trained ViT ancestor models. Based on this structure, it automatically constructs and initializes descendant models of different sizes, significantly reducing training costs and deployment complexity while maintaining model performance. This effectively addresses issues such as the difficulty in unifying the modeling of inter-layer knowledge in Transformers, the ineffective utilization of parameter redundancy, the lack of a systematic mechanism for model expansion, and high downstream transfer costs. It exhibits good interpretability, transferability, and deployability, making it suitable for various visual tasks and resource-constrained scenarios. Experimental results show that the proposed method maintains or even improves model accuracy while reducing the number of parameters by approximately 50%, achieving rapid convergence and good generalization performance. This invention can be widely applied to scenarios such as large-scale visual model compression, knowledge reuse, and cross-task transfer. Attached Figure Description

[0063] Figure 1 A framework diagram of the method provided by this invention for compressing ancestor model knowledge and inheriting it to descendant models through optimal transmission;

[0064] Figure 2This diagram illustrates the comparison of classification accuracy of the offspring model constructed using the weight sharing method described in this invention on multiple downstream image classification datasets, serving to explain the performance of constructing an offspring model based on weight sharing of learning genes in different downstream tasks.

[0065] Figure 3 This diagram illustrates the comparison of classification accuracy of the progeny model constructed using the weight concatenation method described in this invention on multiple downstream image classification datasets. It is used to explain the performance of the progeny model in different downstream tasks when constructing a progeny model based on weight concatenation of learning genes.

[0066] Figure 4 This diagram illustrates the comparison of the training convergence process of the descendant model constructed using the weight sharing method (left figure) and the weight concatenation method (right figure) of this invention on the ImageNet dataset under the same training conditions with a model with randomly initialized parameters. It is used to illustrate the advantages of the descendant model constructed by this invention in terms of training efficiency.

[0067] Figure 5 This is a comparison of the classification accuracy of different models trained on the ImageNet dataset under the same training conditions but with different proportions of training data. It is used to illustrate the performance of the progenitor model constructed in this invention under different data scales. Detailed Implementation

[0068] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0069] This invention is based on a multi-layered visual Transformer ancestor model. It constructs cross-layer feature difference relationships by uniformly representing and aligning the intermediate features of each network layer of the ancestor model. Furthermore, it introduces an entropy-regularized optimal transfer model to learn knowledge transfer weights between layers, thereby extracting learning gene parameters that can be shared across layers and have transferability. Based on this, and according to different application scenarios and resource constraints, it utilizes the learning genes to construct descendant models of different depths or widths. These descendant models can achieve rapid adaptation and efficient inference for downstream tasks without accessing the original training data of the ancestor model. The method provided by this invention for compressing ancestor model knowledge and inheriting it to descendant models through optimal transfer specifically includes the following steps:

[0070] Step S1: In this embodiment, firstly, an ancestor model containing multiple Transformer network layers is constructed, and the input samples are... The ancestor model is input for forward propagation. In the ancestor model's... In each layer, the category label embedding vector corresponding to that layer is extracted. in, The feature dimension of the category label embedding vector and the attention map output by the multi-head self-attention mechanism are represented. ,in Indicates the number of heads of attention. This represents the sequence length and is used to characterize the information dependency pattern across positions. Among them, the category label embedding vector... Attention map is used to reflect the overall semantic features of the input sample at this layer. It is used to describe the information dependencies between different positions in an input sequence.

[0071] In this embodiment, to ensure that features from different sources have a consistent numerical scale and statistical distribution during subsequent cross-layer comparisons, normalization operations are performed on the category label embedding vector and attention map, respectively. The normalization function is denoted as [function name missing]. The normalized features are then concatenated along the feature dimensions to obtain a unified feature representation for this layer.

[0072] ,

[0073] Among the symbols This represents a vector-level concatenation operation. This expression enables the fusion of semantic representation and structural association information in a unified space, mapping intermediate features from different layers of the ancestor model to a unified feature representation space. This lays the foundation for subsequent construction of cross-layer knowledge associations and optimal transmission matching.

[0074] Step S2: After obtaining the unified feature representations of each layer of the ancestor model, in order to establish the knowledge association relationship between different layers of the ancestor model, this embodiment further constructs a difference metric between each layer of the ancestor model and the candidate learning genes, and introduces a trainable learning gene feature vector. ,in, The index number of the learning gene is used to indicate the first... One candidate learning gene, This represents the dimension of the learned gene feature vector. And for the ancestor model... Layer feature representation The difference between the learned gene feature vector and the target gene feature vector is measured. Specifically, in this embodiment, Euclidean squared distance is used as the difference measure, which is defined as follows:

[0075] ,

[0076] A cross-layer feature difference matrix is ​​constructed by calculating the differences between all layers of the ancestor model and the learned genes. This distance metric is used to characterize the relative matching degree of different layers in the ancestor model during the learning gene formation process, and serves as the cost input for subsequent cross-layer knowledge transfer modeling. This distance metric enables a unified comparison of feature representations at different depths in the ancestor model, establishing a necessary numerical foundation for subsequent cross-layer knowledge transfer.

[0077] Step S3, in this embodiment, to automatically learn the optimal correspondence between each layer of the ancestor model and the learning gene from the cross-layer feature difference matrix, the cross-layer knowledge mapping process is modeled as an entropy-regularized optimal transmission problem. Using the cross-layer transmission matrix... As optimization variables, the following objective function is constructed:

[0078] ,

[0079] in This is the entropy regularization coefficient, used to control the smoothness of the transmission distribution; , representing the cost of cross-layer feature matching. Meanwhile, in this embodiment, a mass conservation constraint is applied to the transfer matrix:

[0080] ,

[0081] in and The normalized hierarchical weight distribution vector ensures that the probability conservation principle is satisfied during cross-layer knowledge transfer. This modeling approach can automatically learn the optimal inter-layer mapping that conforms to probabilistic constraints, enabling the learning gene to inherit multi-layer representations from ancestor models.

[0082] Step S4: In this embodiment, the iterative scaling (Sinkhorn) method is used to solve the above-mentioned entropy-regularized optimal transport problem. First, a kernel matrix is ​​constructed based on the cross-layer feature difference matrix and the entropy regularization coefficients:

[0083] ,

[0084] Then, the scaling vector is updated alternately:

[0085] ,

[0086] The final cross-layer knowledge transfer matrix is ​​obtained as follows:

[0087] ,

[0088] Among the symbols This represents element-wise division. After multiple iterations, the transfer matrix converges to a stable state, yielding a cross-layer knowledge transfer matrix that satisfies regularization conditions and probabilistic constraints. This matrix reveals the degree of contribution of each layer of the ancestor model to the construction of learning genes, realizing the structured redistribution of knowledge between layers.

[0089] Step S5: After obtaining the cross-layer knowledge transfer matrix, in this embodiment, the ancestor model parameters are weighted and fused across layers based on the transfer matrix to construct shared learning gene parameters. Let the ancestor model be... The model parameters of the layer are Then the first Each learning gene parameter is represented as:

[0090] ,

[0091] This fusion method can compress repetitive or highly correlated structural patterns between different layers in the ancestor model into a small number of shared parameters, thereby capturing consistent attention patterns across layers and forming a learning gene with good generalization and transferability, achieving compact expression and cross-task transferability of the learning gene.

[0092] Step S6: In this embodiment, based on different deployment environments and according to the computing resource conditions and performance requirements of the target application scenario, progeny models of different scales are constructed based on the learned gene parameters. On the one hand, a depth-tunable model structure can be constructed by reusing the learned gene parameters in multiple network layers, thus creating a depth-scalable architecture; on the other hand, a linear mapping matrix can be introduced. The learning gene parameters are transformed into neural network structure parameters of different dimensions, which take the following form:

[0093] ,

[0094] This allows the learning gene parameters to adapt to network structures of different widths, achieving the ability to generate multiple models based on a unified learning gene. This step enables flexible scalability of the model size in computationally limited scenarios, while maintaining the structured knowledge inherited from ancestor models. Furthermore, this embodiment supports two descendant model expansion methods: weight sharing and weight concatenation, to meet different deployment requirements.

[0095] The construction of the descendant model can choose from the following two expansion strategies according to different task requirements, in order to adapt to different resource conditions and task complexity:

[0096] (1) Weight Sharing Strategy: The learning gene is copied to multiple network layers and the same parameters are shared. The same learning gene parameters are reused in multiple layers to form a lightweight architecture, reduce model size and training overhead, and ensure representation consistency.

[0097] (2) Weight Stitching strategy: By equipping the learning gene with layer-specific transformations, non-shared expansion is performed on layers of different depths, enabling the offspring model to adapt to deeper or structurally different neural network architectures.

[0098] These two strategies are compatible and flexibly selectable, enabling flexible construction of descendant models under varying resource budgets and downstream task complexities to support diverse deployment scenarios. The constructed descendant models can be used not only in lightweight training mode but also in full-parameter fine-tuning. In this mode, all trainable parameters of the network are optimized end-to-end using stochastic gradient descent methods to further enhance the adaptability of the learning genes to specific tasks. To verify the broad applicability of this invention, the descendant models can be trained on various datasets of different sizes and difficulties, including OxfordFlowers-102, CUB-200-2011, Stanford Cars, CIFAR-10, CIFAR-100, Food-101, Tiny-ImageNet, and iNaturalist-2019. Cross-dataset validation further improves the model's generalization ability.

[0099] Step S7: After constructing the descendant model, the descendant model is directly applied to the training and inference process of downstream tasks. This allows for knowledge transfer without accessing the original training data of the ancestor model, thus ensuring data privacy and security. In this embodiment, the descendant model is initialized with the learning gene parameters and trained or fine-tuned on the downstream dataset. Since the learning gene has inherited the cross-layer structural knowledge from the ancestor model, the descendant model can achieve fast and stable convergence in fewer training rounds and obtain high inference accuracy and good generalization performance in different downstream tasks.

[0100] The training of the progenitor model in downstream tasks can also adopt a full-parameter fine-tuning approach, that is, performing end-to-end optimization on all trainable parameters on the network structure initialized by the learning gene, and using stochastic gradient descent-like optimization methods to update all network parameters on the downstream dataset to maximize the adaptability of the learning gene on specific tasks.

[0101] The method of this invention can be applied to image classification. After constructing a descendant model using the method of this invention, it is necessary to use image data to form a dataset, input it into the descendant model constructed by the method of this invention for training, inference, and finally output the image category.

[0102] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a method for optimally transferring ancestor model knowledge compression and inheritance to descendant models.

[0103] This invention is based on the ancestor model of the Visual Transformer (ViT) with a multi-layer network structure. It obtains transferable learning gene parameters by uniformly extracting intermediate features from each layer, establishing inter-layer difference metrics, and utilizing entropy regularization to learn cross-layer knowledge mapping relationships through optimal transfer. Based on these parameters, descendant models of different depths or widths are constructed. These descendant models can quickly complete knowledge transfer on unseen downstream tasks, achieving efficient convergence and accurate inference.

[0104] To verify the broad applicability of the method of this invention, this test case uses a descendant model constructed based on the method of this invention. The model is reconstructed using both weight sharing and weight concatenation methods, and tested on multiple publicly available downstream image classification datasets. These datasets include Oxford Flowers-102, CUB-200-2011, StanfordCars, CIFAR-10, CIFAR-100, Food-101, Tiny-ImageNet, and iNaturalist-2019, covering various application scenarios from coarse-grained classification to fine-grained recognition. The training of the descendant model on downstream tasks is based on datasets of different sizes and difficulties. Cross-dataset validation further improves the model's generalization ability.

[0105] Figure 2 The classification accuracy of the descendant model built based on weight sharing is shown on different downstream datasets. Figure 3 This demonstrates the classification performance of a descendant model built using a weight concatenation method on the same dataset. Figure 2 and Figure 3 It can be seen that in most downstream tasks, the progeny model constructed in this invention has a classification accuracy that is better than or close to that of the model obtained by the traditional fine-tuning method under the same or fewer parameter scale, indicating that parameter initialization by learning genes can effectively improve the model's adaptability to different tasks.

[0106] To further verify the advantages of this invention in model training efficiency, the ImageNet dataset was selected as the downstream task in this test case to compare the convergence process of the descendant model constructed by the method of this invention with that of a randomly initialized parameter model under the same training conditions. In the experiment, the descendant model was constructed using weight sharing and weight concatenation methods respectively, while keeping other training parameters consistent.

[0107] like Figure 4As shown, the left figure compares the training processes of the progeny model built based on weight sharing and the randomly initialized model, while the right figure compares the training processes of the progeny model built based on weight concatenation and the randomly initialized model. It can be seen that, under the condition of achieving similar classification accuracy, the progeny model constructed in this invention requires significantly fewer training epochs than the randomly initialized model, and the training convergence speed is improved by approximately 3.3 to 4.0 times. This indicates that the learning genes extracted in this invention can provide good parameter initialization for the progeny model, effectively accelerating the model training process.

[0108] In this test case, the performance of the progenitor model constructed according to this invention is further examined under different training data scales. Specifically, on the ImageNet dataset, using 100%, 50%, and 25% of the training data respectively, the classification performance of the randomly initialized model, the traditional fine-tuned model, and the progenitor model constructed based on the method of this invention are compared.

[0109] Figure 5 Test results for different models under the aforementioned training data proportions are presented. Figure 5 As can be seen, when training data is sufficient, the progeny model constructed in this invention achieves classification accuracy comparable to traditional methods. However, when training data is limited, the progeny model constructed based on this invention still maintains high classification performance, with a significantly improved accuracy compared to randomly initialized models and traditional fine-tuning models, especially under 25% data scale conditions. This demonstrates that this invention can effectively improve the model's generalization ability and data utilization efficiency in scenarios with few samples.

[0110] The test results above show that by extracting cross-layer learning genes from ancestral models and using them to construct offspring models, this invention can not only maintain high classification accuracy in various downstream tasks, but also significantly accelerate model training convergence speed. Furthermore, it demonstrates stronger robustness and generalization ability even with limited training data, thus verifying the effectiveness and practical value of the technical solution of this invention.

[0111] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method of compressing and inheriting ancestor model knowledge to descendant models through optimal transfer, characterized by, Includes the following steps: Step S1: Construct an ancestor model containing multiple Transformer network layers and obtain a unified feature representation for each layer of the ancestor model; Step S2: Construct a measure of the difference between each layer of the ancestor model and the candidate learning genes; Step S3: Model the cross-layer knowledge mapping process as an entropy-regularized optimal transmission problem; Step S4: Solve the entropy regularization optimal transmission problem to obtain the cross-layer knowledge transmission matrix; Step S5: Perform cross-layer weighted fusion of ancestor model parameters based on the cross-layer knowledge transfer matrix to construct shared learning gene parameters; Step S6: Construct offspring models of different scales based on the learned gene parameters; Step S7: Apply the completed descendant model to the training and inference of downstream tasks.

2. The method of claim 1, wherein, Step S1 specifically includes: An ancestor model comprising a plurality of Transformer network layers is constructed, and an input sample is input into the ancestor model forward propagation is performed in the input ancestor model; in the first layer of the ancestor model , a category mark embedding vector corresponding to the layer is extracted respectively and an attention map output by a multi-head self-attention mechanism , wherein denotes the number of attention heads, denotes the sequence length, is a category mark embedding vector, denotes the feature dimension of the category mark embedding vector, is an attention map; Normalization is performed on the category label embedding vector and attention map respectively, and the normalized features are concatenated along the feature dimensions to obtain the first... The unified characteristics of a layer are represented as follows: ， The symbols This represents a vector-level concatenation operation. This is the normalization function.

3. The method for compressing ancestor model knowledge and inheriting it to descendant models through optimal transmission according to claim 1, characterized in that, Step S2 specifically includes: For the ancestor model Layer feature representation Feature representation of candidate learning gene layers Perform a difference measurement, where Initialize vectors for trainable, learnable gene features. Indicates the index number of the learning gene. The dimension of the learned gene feature vector is represented; the Euclidean squared distance is used as the difference measure, and the calculation formula is as follows: ， A cross-layer feature difference matrix is ​​constructed by calculating the differences between all layers of the ancestor model and the learned genes. .

4. The method for compressing ancestor model knowledge and inheriting it to descendant models through optimal transmission according to claim 1, characterized in that, Step S3 specifically includes: With cross-layer transmission matrix To optimize the variables, construct the objective function: ， in, Here is the entropy regularization coefficient. This represents the cost of cross-layer feature matching; simultaneously, a mass conservation constraint is applied. ， in This is a normalized hierarchical weight distribution vector to ensure that cross-layer transmission satisfies probability conservation.

5. The method for compressing ancestor model knowledge and inheriting it to descendant models through optimal transmission according to claim 1, characterized in that, Step S4 employs an iterative scaling method to solve the entropy-regularized optimal transmission problem, specifically including: First, construct the kernel matrix based on the cross-layer feature difference matrix and the entropy regularization coefficient: ， Then, the scaling vector is updated alternately: ， The final cross-layer knowledge transfer matrix is ​​obtained as follows: ， The symbols This indicates element-wise division; After multiple iterations until convergence, a cross-layer knowledge transfer matrix satisfying the regularization condition and probability constraints is obtained. .

6. The method for compressing ancestor model knowledge and inheriting it to descendant models through optimal transmission according to claim 1, characterized in that, Step S5 specifically includes: When the ancestor model parameters are attention weight parameters, let the ancestor model be... The attention weights of the layer are Then the first Each learning gene parameter is represented as: 。 7. The method for compressing ancestor model knowledge and inheriting it to descendant models through optimal transmission according to claim 1, characterized in that, The construction of descendant models of different scales in step S6 includes any of the following methods: (1) Reuse the learned gene parameters in multiple network layers to construct a deep adjustable model structure; (2) The learning gene parameters are transformed into neural network structure parameters of different dimensions. The calculation method is as follows: ， in, It is a linear mapping matrix.

8. The method for compressing ancestor model knowledge and inheriting it to descendant models through optimal transmission according to claim 1, characterized in that, In step S7, the training of the offspring model in the downstream task adopts a full parameter fine-tuning method. End-to-end optimization is performed on all trainable parameters on the network structure initialized by learning the gene, and all network parameters are updated on the downstream dataset using a stochastic gradient descent-like optimization method.

9. An image classification method, comprising: Collect image data and form a dataset; The feature is that the dataset is input into a descendant model constructed by the method of compressing ancestor model knowledge and inheriting it to the descendant model through optimal transmission as described in any one of claims 1-8, and the model is trained, inferred, and outputs image classification results.

10. An apparatus for compressing and inheriting ancestor model knowledge to descendant models through optimal transmission, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes a computer program to implement the steps of the method of optimal transfer of ancestor model knowledge and its inheritance to descendant models as described in any one of claims 1-8.

Images