Method for de-homogenizing large models based on quality diversity balancing and dynamic architecture
By generating unique anchor vectors for model features and implementing dynamic constraint mechanisms, the problem of feature homogenization during the training and fine-tuning of large models is solved, achieving stable and differentiated expression of the model throughout its entire lifecycle.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNICOM ONLINE INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2026-03-31
- Publication Date
- 2026-07-31
AI Technical Summary
Existing large models are prone to convergence of feature space and output distribution during training and multiple rounds of fine-tuning, making it difficult to maintain stable and differentiated expression throughout their entire lifecycle.
By generating unique anchor vectors for model features, and combining domain-specific differential encoding of multi-source heterogeneous data, dynamic feature evolution differentiation, and feature-output-specific decoupling mapping, a fixed reference benchmark for the underlying feature space is established, and dynamic constraints are applied during training and fine-tuning to ensure the differentiation of feature distribution.
It effectively prevents feature homogenization in large models during the later stages of training and multiple rounds of fine-tuning, ensuring the model's differentiated expressive ability in the long term and avoiding output regression homogenization.
Smart Images
Figure CN122490115A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a method for dehomogenizing large models based on quality diversity balancing and dynamic architecture. Background Technology
[0002] With the development of deep learning technology, large-scale models have been widely applied in various vertical fields. Typically, to adapt a basic large-scale model to a pre-defined task, researchers continuously pre-train or fine-tune the model using business data from the relevant field. However, because most mainstream large-scale models currently employ similar underlying network architectures and the same optimization objectives, after multiple rounds of training, different models tend to exhibit high convergence in feature representation space and output probability distribution, resulting in homogenization. This homogenization leads to models trained by different vendors or business lines tending to generate highly similar inference results in terms of structure and semantics when faced with the same input, losing the differentiated characteristics that a business model should possess.
[0003] Current conventional solutions to the model homogenization problem mainly rely on introducing random noise during the training phase or imposing static regularization penalties in the loss function. While these methods can increase the randomness of model parameter updates in the early stages of training, as the number of training epochs increases and the model converges towards the optimal solution, static regularization constraints struggle to counteract the convergence inertia at the lower levels, resulting in an inability to effectively suppress feature homogenization in the later stages of training. Furthermore, existing large models typically employ a shared linear mapping network structure in the output decoding phase. This universal mapping mechanism cannot fully transmit subtle differences in features within the model to the final output. Moreover, when the model is subsequently deployed in real-world applications and fine-tuned multiple times with new data, the shallow differentiated features established early on are easily overwritten by gradient updates from new data during backpropagation, causing the model output to revert to homogenization and failing to maintain stable differentiated representation capabilities throughout the model's lifecycle. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention provides a method for dehomogenizing large models based on quality diversity balancing and dynamic architecture. This method solves the problem that existing large models tend to homogenize their feature space and output distribution during continuous training and multiple rounds of fine-tuning, and it is difficult to maintain stable and differentiated expressions throughout the model's entire lifecycle.
[0005] To achieve the above objectives, this invention provides the following technical solution: a large-scale model dehomogenization method based on quality diversity balancing and dynamic architecture, comprising the following steps:
[0006] The system utilizes a multi-source heterogeneous knowledge input unit to preprocess the received multi-source heterogeneous data and generate knowledge type labels based on data attributes. In the preprocessing stage, for text data, a byte-pair encoding algorithm is used for word segmentation to generate a sequence of tokens; for knowledge graph data, entity linking and relation extraction are performed to generate a sequence of triples, which are then vectorized; for numerical and structured data, normalization processing is performed, mapping the numerical values to a preset interval and generating structured feature vectors.
[0007] A normalized, high-dimensional random vector is generated using the global constraint unit for unique anchor points of model features. This vector serves as the unique anchor point vector for model features and is embedded into the parameter initialization process of subsequent processing units. Specifically, a high-dimensional random vector whose elements follow a standard normal distribution is generated using a random number generator, with its dimension set according to the hidden layer dimension of the base model. The magnitude of the high-dimensional random vector is calculated, and each element is divided by this magnitude to generate a unique anchor point vector for model features with a magnitude of 1.
[0008] Domain-specific differential encoding units are used to construct non-overlapping dedicated encoding spaces based on knowledge type labels. Combined with the unique anchor vectors of model features, domain-specific differential encoding is performed on the preprocessed data to generate knowledge encoding vectors. The total dimension of the embedding layer of the base model is obtained, and the total dimension of the embedding layer is divided into multiple subspaces based on the number of knowledge type labels. A non-overlapping subspace with indexes is assigned to each knowledge type as its dedicated encoding space. Dedicated encoding functions are used to encode each type of knowledge; the calculation formula is as follows: ; in, For the first The specific encoding result of class knowledge, For dedicated encoding weight matrix, For input data The initial embedding vector, For bias terms, The anchor point fusion coefficient, This serves as a unique anchor vector for the model's features. Subsequently, the individual encoding results are merged into a unified knowledge encoding vector using a masking mechanism.
[0009] The knowledge encoding vector is input into the base model. The dynamic feature evolution differentiation unit calculates the feature similarity change rate during the base model training process to determine the feature evolution stage. Based on the feature evolution stage and the model's unique anchor vector, phased dynamic feature differentiation constraints are applied to the hidden layer feature vectors of the base model. The cosine similarity difference between the hidden layer feature vectors of the current training epoch and the previous training epoch is calculated as the feature similarity change rate. If the feature similarity change rate is greater than 0.05, it is determined to be in the rapid evolution stage; if the feature similarity change rate is between 0.01 and 0.05, it is determined to be in the stable evolution stage; if the feature similarity change rate is less than 0.01, it is determined to be in the convergence stage. The corresponding constraint coefficients are matched according to the determined feature evolution stage, with the constraint coefficients for the rapid evolution stage, stable evolution stage, and convergence stage increasing sequentially. The calculation formula for applying dynamic feature differentiation constraints is: ; in, For the constrained eigenvectors, For hidden layer feature vectors, This is the constraint coefficient. Simultaneously, the feature differentiation degree is calculated, and the constraint coefficient is automatically increased when the feature differentiation degree falls below a set threshold.
[0010] The constrained feature vectors are input into a dedicated decoupling mapping network using a feature-output dedicated decoupling mapping unit. This is combined with the output style-unique encoding vector generated by transforming the model's unique feature anchor vectors to produce a differentiated output prediction vector. The constrained feature vectors are then processed by a parameter-independent two-layer fully connected network. A linear transformation is performed on the model's unique feature anchor vectors using the style encoding weight matrix to generate the output style-unique encoding vector. The formula for calculating the differentiated output prediction vector is as follows: ; in, To differentiate the output prediction vector, This represents the forward computation process of a two-layer fully connected network. The style encoding weight matrix.
[0011] After the processing steps of domain-specific differential encoding, phased dynamic feature differentiation constraints, and generation of differential output prediction vectors, full-link correlation calibration is performed. The cosine similarity between the current result vector and the model's unique feature anchor vector is calculated. If the cosine similarity is less than 0.8, a linear transformation is performed on the result vector to improve its similarity to the model's unique feature anchor vector.
[0012] Regarding the protection of the model throughout its entire runtime, the embedding state of the unique anchor vector of the model features within the system architecture is maintained during the inference phase after model training convergence and during multiple rounds of fine-tuning. During multiple rounds of fine-tuning, the value of the unique anchor vector of the model features remains fixed and is not updated with the backpropagation algorithm.
[0013] This invention provides a method for dehomogenizing large models based on quality diversity balancing and dynamic architecture. It offers the following advantages: 1. This invention generates a unique anchor vector for model features and embeds it into the parameter calculation stages of data domain differentiation coding, feature evolution constraints, and output decoupling mapping. This establishes a fixed reference benchmark in the underlying feature space of the model and forces the feature distribution of the basic model to shift in the direction determined by the anchor vector. This avoids the feature homogenization phenomenon that occurs when multiple large models process the same task from the underlying architecture.
[0014] 2. This invention introduces a dynamic feature evolution differentiation mechanism, which determines the current feature evolution stage by calculating the feature similarity change rate during model training, and adaptively adjusts the constraint coefficients on the hidden layer feature vectors accordingly. In the later stage when the model training tends to converge, the differentiation constraint intensity is increased, which overcomes the technical defect of traditional static regularization methods in that they are difficult to effectively suppress the convergence of model features in the later stage of training.
[0015] 3. This invention employs a dedicated mapping network decoupled from the basic model, combined with output style-unique encoding to generate prediction vectors. It also incorporates an anchor parameter freezing strategy during the full-link correlation calibration and multi-round fine-tuning stages. The dedicated decoupled mapping cuts off the output distribution consistency caused by the general decoding network, while the full-link calibration and anchor parameter freezing ensure that the differential features at the bottom layer of the model remain stable during long-term multi-round iterations and downstream fine-tuning, preventing the established differential features from being covered by new domain data. Attached Figure Description
[0016] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a flowchart illustrating the implementation of the present invention; Figure 3 This is a schematic diagram of the feature evolution stage determination and dynamic constraint mechanism of the present invention; Figure 4 This is a logic diagram for the domain-specific differential encoding and feature-output mapping of the present invention. Detailed Implementation
[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] See attached document Figure 1 This invention provides a method for dehomogenizing large models based on quality diversity balance and dynamic architecture adaptation. The system architecture on which this method depends includes: a multi-source heterogeneous knowledge input unit, a domain-specific differential encoding unit, a dynamic feature evolution differentiation unit, a feature-output dedicated decoupling mapping unit, and a global constraint unit for unique anchor points of model features.
[0019] The above units are connected sequentially according to the data processing link. The output of the multi-source heterogeneous knowledge input unit is connected to the input of the domain-specific differential coding unit. The output of the domain-specific differential coding unit is connected to the input of the basic model. The hidden layer features of the basic model are connected to the dynamic feature evolution and differentiation unit. The output of the dynamic feature evolution and differentiation unit is connected to the input of the feature-output dedicated decoupling mapping unit. The model feature unique anchor point global constraint unit establishes data connections with the domain-specific differential coding unit, the dynamic feature evolution and differentiation unit, and the feature-output dedicated decoupling mapping unit, respectively, to provide a unified anchor point vector. Each processing unit uses this anchor point vector to perform correlation calibration of the processing results.
[0020] See attached document Figure 2 The method for dehomogenizing large models specifically includes the following steps: The multi-source heterogeneous knowledge input unit receives input data including text data, knowledge graph data, numerical data, and structured data, and performs preprocessing operations for the corresponding data types. After preprocessing, the multi-source heterogeneous knowledge input unit classifies all preprocessed data into knowledge types, generates corresponding type labels, records the dimensional information of each type of data, and outputs a standardized classification dataset.
[0021] The Global Constraint Unit for Unique Anchor Points of Model Features generates a high-dimensional random vector. This unit normalizes the generated high-dimensional random vector to obtain an initial anchor vector. This anchor vector is then embedded into the parameter initialization process of the Domain Differentiation Encoding Unit, the Dynamic Feature Evolution Differentiation Unit, and the Feature-Output Decoupling Mapping Unit.
[0022] The domain-specific differential coding unit acquires a standardized classification dataset and anchor vectors. Based on the knowledge type label, the unit assigns an independent coding space to each knowledge type, with no overlap in the dimensions of the coding spaces. The unit then performs a dedicated coding operation on each knowledge type using the anchor vectors, generating a coding vector that falls within the corresponding dedicated coding space. Finally, the unit merges the coding vectors from all types to output a unified knowledge coding vector.
[0023] The knowledge encoding vector is input into the base model for initial training. The dynamic feature evolution differentiation unit acquires feature information during the training process of the base model and calculates the feature similarity change rate. The dynamic feature evolution differentiation unit identifies the feature evolution stage of the base model training based on the feature similarity change rate. The feature evolution stage is divided into a rapid evolution stage, a stable evolution stage, and a convergence stage.
[0024] The dynamic feature evolution differentiation unit calls upon the corresponding constraint coefficients based on the identified feature evolution stage. Combining the anchor vector and constraint coefficients, the dynamic feature evolution differentiation unit calculates the hidden layer feature vectors of the base model, generating dynamically constrained differentiated feature vectors. Simultaneously, the dynamic feature evolution differentiation unit calculates the feature differentiation degree and adjusts the constraint coefficients when the feature differentiation degree falls below a set threshold.
[0025] The Feature-Output Decoupled Mapping Unit (FAMA) acquires dynamically constrained differential feature vectors. These vectors are then fed into a fully connected network containing hidden layers. The parameters of this fully connected network are independent of the base model and initialized based on anchor vectors. The FAMA performs a linear transformation on the anchor vectors to generate an output style-unique encoding vector. Finally, the FAMA adds the output vector from the fully connected network to the output style-unique encoding vector to generate the output prediction vector.
[0026] After each processing step of the domain-specific differential encoding unit, dynamic feature evolution differentiation unit, and feature-output dedicated decoupling mapping unit, correlation calibration is performed. The cosine similarity between the result vector generated in the current processing step and the anchor vector is calculated. If the cosine similarity is lower than a set value, a linear transformation is applied to the result vector. After calibration, the loss value is calculated based on the output prediction vector and the true label, and the parameters of the corresponding unit are updated via backpropagation. The steps of encoding, evolution constraint, mapping, and calibration are repeated until the model training converges.
[0027] During the inference phase after model training convergence and in the multiple rounds of fine-tuning, the anchor vectors are maintained in their embedded state within the system architecture. The anchor vectors do not participate in the model parameter update process during this phase, thus maintaining the global constraint state of the model features.
[0028] The model's unique anchor point global constraint unit utilizes a random number generator to produce a high-dimensional random vector. The dimension of this high-dimensional random vector... The configuration is based on the hidden layer dimension of the base model. In a specific implementation scenario where the hidden layer dimension of the base model is 2048, the following will be implemented: The value is set to 512 to ensure that the anchor vector has sufficient representational capacity. All elements in the high-dimensional random vector follow a standard normal distribution. Independent sampling is performed to ensure the randomness and diversity of the vector's spatial distribution.
[0029] After obtaining the initial random vector, the global constraint unit for unique anchor points of the model features performs normalization on it. This is achieved by calculating the magnitude of the vector and dividing each element of the vector by the magnitude, resulting in the final unique anchor point vector. The modulus is equal to 1. This normalization operation provides a unified metric for cosine similarity calibration of subsequent processing units, eliminating the imbalance of constraint strength caused by differences in vector magnitude. The generated... As the unique feature identifier of the model, it is distributed by the model feature unique anchor point global constraint unit to the system's domain-specific differential coding unit, dynamic feature evolution differentiation unit, and feature-output dedicated decoupling mapping unit. During the system parameter initialization phase, each unit will... It is embedded into the initial state of its weight matrix. Specifically, the system uses a linear transformation matrix to... The parameters are projected onto the parameter space corresponding to each unit and fused with the randomly initialized weight parameters, thereby completing the parameter binding of each processing module to the unique anchor vector at the underlying logic level.
[0030] During model training, the unique anchor vector It serves as a reference benchmark for global constraints. The global constraint unit, a unique anchor point for model features, continuously monitors the intermediate feature vectors generated by each layer of the model and calculates the intermediate feature vectors and their corresponding values. The spatial topological relationships between them force the model features to evolve in a preset direction of differentiation. Because It is generated during the initialization phase and has a very high probability of non-overlap between different models. This mechanism ensures that even with the same training data and task objectives, different models will converge to different feature subspaces due to their unique anchor points, thereby blocking the path of feature cross-convergence at the underlying level.
[0031] This global constraint mechanism not only plays a role in the early stages of model building but also persists throughout the model update and fine-tuning cycles. During multiple rounds of fine-tuning, the globally constrained unit, representing the unique anchor point of the model's features, is locked. The value of the anchor point is set so that it does not require parameter updates during backpropagation. Through this non-volatile anchor point constraint, the model absorbs new knowledge while its core feature distribution remains strongly correlated with the unique anchor point vector, avoiding the loss of differentiated features due to continuous fine-tuning.
[0032] The multi-source heterogeneous knowledge input unit performs standardized preprocessing operations on received data of different dimensions and modalities. For text data in fields such as industrial quality inspection, the multi-source heterogeneous knowledge input unit calls the byte encoding algorithm to perform word segmentation, constructs a word sequence by recognizing sub-word units, and removes non-printable characters and redundant spaces from the data stream. For graph data such as medical knowledge graphs, the multi-source heterogeneous knowledge input unit performs entity recognition and relation extraction, transforming unstructured graph relations into a triple sequence of head entity, relation, and tail entity, and performing initial vectorization mapping on it. For financial numerical or government structured tabular data, the multi-source heterogeneous knowledge input unit performs normalization mapping, mapping the original numerical values to the [0,1] interval, and transforming them into fixed-dimensional structured feature vectors.
[0033] After completing the above preprocessing, the multi-source heterogeneous knowledge input unit assigns a corresponding knowledge type label to each piece of data based on its data source and logical attributes. Knowledge type tags include text tags. Atlas Tags Numerical labels and structured tags The multi-source heterogeneous knowledge input unit simultaneously records the total amount and initial dimensionality information of each type of dataset, and outputs a standardized classification dataset containing type indexes, providing data support for subsequent domain-specific encoding.
[0034] Domain-specific differential coding units construct their own unique coding spaces with non-overlapping dimensions based on the knowledge type labels in the standardized classification dataset. The total dimension of the embedding layer of the base model is set to [value missing]. The domain-specific differential coding unit divides the total dimension into several subspaces, the number of which corresponds to the number of knowledge type tags. In an implementation scenario, if... With 2048 knowledge points and 4 knowledge types, the domain-specific differentiated coding unit allocates a dedicated 512-dimensional coding subspace for each knowledge type. These subspaces are strictly isolated in terms of dimensional indices; for example, the text coding space occupies indices [0, 511], the graph coding space occupies indices [512, 1023], the numerical coding space occupies indices |1024, 1535|, and the structured coding space occupies indices |1536, 2047|. This physical dimensional isolation ensures that different types of knowledge do not overlap in dimensions during the feature representation stage, thus achieving differentiation in knowledge representation at the underlying logical level.
[0035] The domain-specific coding unit calls a dedicated domain-specific coding function for each type of knowledge. This encoding function will take the input data. It is mapped to a pre-defined dedicated encoding space. The specific calculation process is as follows: ; in, For the corresponding to the first A dedicated encoding weight matrix for class-specific knowledge. For input data The preprocessed initial embedding vector, This is a learnable bias vector. Parameters Anchor point fusion coefficient, used to adjust the unique anchor point vector. The influence strength on the encoding result is set between 0.1 and 0.3. The domain-specific differential encoding unit uses this function to linearly fuse general knowledge features with the model's unique anchor features, ensuring that the generated encoding result not only contains the original knowledge semantics but also possesses the feature distribution imprint unique to the model.
[0036] After each type of knowledge is processed by its dedicated encoding function, the resulting encoding vectors are confined to their respective dimensional ranges. The domain-specific differential encoding unit then performs a cross-domain knowledge differential fusion operation. This operation integrates the encoding vectors falling into different subspaces into a unified knowledge encoding vector through tensor concatenation or weighted summation. During the fusion process, the domain-specific differential coding unit uses a masking mechanism to shield interference information outside each subspace, while retaining the differential characteristics of each dimension interval.
[0037] The generated knowledge encoding vector It needs to be a unique anchor vector of the model features. Perform correlation calibration. Calculate the domain-specific differential coding unit. and The cosine similarity. If the calculated similarity value is lower than 0.8, the domain-specific differential coding unit will... Perform linear transformation calibration. The specific calibration method is to... Towards The system performs projection compensation on the vector directions until the similarity between the two is increased to above 0.8. Through this calibration step, the system ensures that the input knowledge encoding already possesses differentiated features strongly bound to the model's identity before entering the basic model training, laying the foundation for subsequent feature evolution and differentiation.
[0038] Dynamic feature evolution differentiation unit obtains the basic model in the current training round Hidden layer feature vectors The dynamic feature evolution differentiation unit calculates the cosine similarity difference between the features of the current round and the features of the previous round to obtain the feature similarity change rate. This rate of change reflects the evolution rate of the model's feature space during training. The dynamic feature evolution differentiation unit is based on... The value is determined using the stage decision function. The training process is divided into three stages.
[0039] when At that time, it was determined to be in a rapid evolutionary stage. At this stage, the model features are in a state of rapid divergence. When At that time, it was determined to be a stable evolutionary stage. During this stage, the changes in model features tend to level off. When the convergence phase is reached, it is determined to be in the convergence phase. At this stage, the model features tend to converge rapidly towards the general optimal solution. Stage determination function. The logical expression is as follows: ; Dynamic feature evolution differentiation units based on the identified stages For the hidden layer feature vector Applying a phased dynamic feature differentiation constraint function This function introduces constraint coefficients. and the unique anchor vector of model features This involves targeted intervention on the original feature distribution. The calculation logic of the dynamic constraint function is as follows: ; Among them, the constraint coefficient Dynamic adjustments are made based on the evolutionary stage. During the rapid evolutionary stage... Dynamic feature evolution differentiation unit setting Weak constraints are applied to preserve the natural divergence of features during the stationary evolution phase. ,set up Moderate-strength constraints are applied to suppress the initial convergence of the features. During the convergence phase... ,set up Strong constraints are applied to prevent features from collapsing into a homogeneous general space.
[0040] The dynamic feature evolution differentiation unit simultaneously performs real-time monitoring of feature differentiation degree. Feature differentiation degree Defined as the current model feature vector The complement of the cosine similarity between the feature vectors and the general base model, with values ranging from [0,1]. The dynamic feature evolution differentiation unit is obtained by subtracting the cosine similarity between the two from 1. .
[0041] When the characteristic differentiation degree is detected At this time, the dynamic feature evolution differentiation unit triggers a feedback adjustment mechanism. This mechanism automatically increases the constraint coefficient of the current stage. To enhance the differentiation guidance, the increase is set to 0.1. Through this closed-loop feedback, the dynamic feature evolution differentiation unit ensures that the model maintains a preset feature differentiation level throughout the entire training cycle, generating dynamically constrained differentiated feature vectors. .
[0042] The aforementioned dynamic constraint mechanism transforms static feature regularization into an adaptive guidance process that dynamically changes with training. By increasing the anchor point constraint weights in the later stages when the model evolution rate decreases and the convergence trend strengthens, this method forces the model feature distribution to shift towards a unique anchor point. Defined subspace offset. This directional evolution ensures that different models can maintain stable differences in underlying features after a full training cycle, solving the problem that traditional static constraint methods cannot suppress feature convergence in the later stages of training.
[0043] See attached document Figure 3 The feature-output dedicated decoupling mapping unit obtains dynamically constrained differential feature vectors. To avoid the convergence of output probability distributions in large models during the decoding stage due to the use of general linear mappings and the Softmax operator, a feature-output dedicated decoupling mapping unit constructs a dedicated mapping network decoupled from the base model. This dedicated mapping network structurally employs a lightweight fully connected neural network (MILP) containing two hidden layers. The input dimension of the dedicated mapping network is set to the hidden layer dimension of the base model, and the output dimension is set to the vocabulary dimension of the base model.
[0044] The parameters of this dedicated mapping network are independent and not shared with the decoding layer parameters of the base model. During parameter initialization, the feature-output dedicated decoupling mapping unit utilizes the unique anchor vector of the model features. Initialize the weight matrix of the dedicated mapping network. This is done by... The spatial distribution characteristics are mapped to the initial state of the weight matrix. The feature-output dedicated decoupling mapping unit ensures that even if the input feature vector is the same, the mapping network of different models will produce different logits, thus blocking the path of output homogenization from the mapping logic level.
[0045] In the output layer of the dedicated mapping network, the feature-output dedicated decoupling mapping unit introduces an output style-unique encoding mechanism. This mechanism uses the anchor vector... A linear transformation is performed to generate a model-specific, style-unique output encoding vector. Specifically, the system utilizes the style encoding weight matrix. right Perform the calculation. The final differentiated output prediction vector. The calculation logic is as follows: ; In this formula, This represents the forward computation process of the dedicated decoupling mapping network. By adding the style-unique encoded vector to the output of the mapping network, the feature-output dedicated decoupling mapping unit implants a non-replicable feature bias into the final predicted probability distribution. This bias ensures that the model's output style remains consistent with the distribution defined by the anchor vector when generating text or performing prediction tasks.
[0046] Feature-output dedicated decoupling mapping unit in generation Then, the associated calibration operation is performed again. The system calculates... and The cosine similarity between them is used. If the similarity is less than 0.8, it is adjusted by linear transformation. The direction of the component, so that it is directed towards The direction is shifted until the similarity threshold requirement is met. This calibration process ensures that the output prediction results are semantically accurate and strictly adhere to the model's unique identity constraint.
[0047] Through the synergistic effect of the decoupled mapping and style encoding described above, this invention achieves end-to-end differentiation from bottom-level intermediate features to top-level predicted output. The dedicated mapping network uses independent parameters to sever its association with the general decoding paradigm, while style-unique encoding further strengthens the differentiated features at the output end. This mechanism effectively solves the problem of model output easily reverting to homogeneity after multiple rounds of fine-tuning, ensuring the output uniqueness of the large model throughout its entire lifecycle.
[0048] After each processing stage—domain-specific differential encoding of multi-source heterogeneous knowledge, dynamic feature evolution differentiation, and feature-output specific decoupling mapping—the system performs an anchor point association calibration operation. The calibration process is led by a global constraint unit for unique anchor points of model features, aiming to ensure that the intermediate variables generated at each stage are uniquely identified by the model. Maintaining high feature consistency. The calibration process first extracts the result vector generated in the current processing step and calculates the unique anchor vector of the model features. Cosine similarity between them.
[0049] When the calculated cosine similarity value is below 0.8, the system determines that the current feature distribution is at risk of shifting towards the universal solution and immediately triggers a linear transformation calibration procedure. This procedure adjusts the orientation of the result vector in the feature space by performing a linear space transformation on the result vector, directing it towards... The directional axis is brought closer. Specifically, through weighted compensation of vector components, the overlap between the result vector and the anchor vector is forcibly increased until the cosine similarity reaches a preset level of 0.8 or higher again. This gated calibration mechanism, distributed at each node of the entire link, ensures that the differential features do not decay when flowing within the model.
[0050] After completing the calibration of each node in the entire link, the system uses the output prediction vector generated by the feature-output dedicated decoupling mapping unit. The model loss value is calculated by comparing it with real labeled data. The choice of loss function includes, but is not limited to, the cross-entropy loss function. Based on the calculated loss value, the system updates the learnable parameters in the domain-differentiated coding unit, the base model layer, the dynamic feature evolution differentiation unit, and the feature-output dedicated decoupling mapping unit through the backpropagation algorithm. During iterative training, the system repeatedly executes the loop of coding differentiation, dynamic constraints, decoupling mapping, and anchor point calibration until the model loss value tends to stabilize or reaches the preset number of training rounds, ultimately producing a differentiated large model with non-replicable feature identifiers.
[0051] This invention achieves differentiated protection throughout the entire model lifecycle through global embedding of anchor vectors. Even after the model enters the inference phase, the system maintains the aforementioned end-to-end processing flow. The multi-source heterogeneous knowledge input unit, the domain-specific differentiated encoding unit, the dynamic feature evolution differentiation unit, and the feature-output dedicated decoupling mapping unit all remain active, and the model features have unique anchor vectors. It is continuously embedded in the computational logic of each stage. Because the input data in the inference process needs to undergo domain-specific encoding and dynamic feature constraints, the final output inference result can always maintain the unique expression style and feature distribution of the model.
[0052] In subsequent rounds of fine-tuning targeting the domain, this invention provides an anti-regression guarantee mechanism. During the fine-tuning process, although some model parameters are updated based on new domain data, the model's unique feature anchor vector remains unchanged. Set to non-update state, i.e. The value remains constant during backpropagation. As a global constraint anchor point, Continuously apply forced guidance to the feature evolution during the fine-tuning process to ensure that while the model learns new domain knowledge, the underlying differentiated feature identifiers will not be washed away or covered by new data.
[0053] By employing end-to-end calibration and non-volatile constraints throughout the lifecycle, this invention addresses the technical challenge of traditional dehomogenization schemes failing after long-term application or multiple rounds of fine-tuning. The unique anchor vector serves not only as a guiding identifier during initialization but also as a long-term physical constraint, solidifying differentiated features within the model's underlying logic. This ensures the deep dehomogenization effect of large models across different application environments and evolutionary stages.
[0054] See attached document Figure 4 This invention provides an adaptive alternative implementation of a large model dehomogenization method, which offers modular alternative logic for different computing power environments, model architectures and security requirements.
[0055] In implementation scenarios involving low computing power environments or edge-side models with a large number of small parameters, this invention provides a lightweight alternative implementation for the knowledge encoding module. In this alternative, the domain-specific differential encoding unit replaces the dedicated encoding space with non-overlapping dimensions with a weighted encoding space where dimensions partially overlap. To maintain differentiated representation across overlapping dimensions, the domain-specific differential encoding unit introduces an overlapping region mask. And set exclusive weight coefficients for different types of knowledge encoding. In this scenario, the encoding function is adjusted as follows: ; in, This method identifies and locks down the common dimensions of the encoding space for different knowledge types. The domain-specific differential encoding unit only performs differential weighting on the feature values that fall on the common dimensions. This scheme sacrifices the absolute isolation of some dimensions in exchange for lower memory overhead, reducing computational resource consumption while ensuring the dehomogenization effect of knowledge encoding by utilizing weight differences.
[0056] In implementation scenarios involving novel large-scale model architectures with unknown training evolution patterns, this invention provides an adaptive alternative implementation for the feature evolution module. In this alternative, the dynamic feature evolution differentiation unit abandons the fixed three-stage decision logic based on a preset rate of change threshold and instead adopts an adaptive constraint mechanism based on feature distribution. The dynamic feature evolution differentiation unit calculates the clustering dispersion of the current model's hidden layer feature set in real time. The clustering dispersion measures the density of the feature vector distribution in space.
[0057] When the calculated feature clustering dispersion decreases, it indicates that the model features tend to concentrate in the preset region, increasing the risk of convergence. The dynamic feature evolution differentiation unit automatically increases the constraint strength accordingly. The strength of this adaptive constraint function is inversely proportional to the feature clustering dispersion; that is, the smaller the dispersion, the greater the guiding force applied between the feature vector and the anchor vector. Through this feedback adjustment, the dynamic feature evolution differentiation unit can automatically adapt to the feature evolution paths of different model architectures without requiring manual pre-setting of stage parameters, thus achieving differentiated evolution support for unknown model architectures.
[0058] In implementation scenarios with high security requirements, such as government affairs, finance, or national defense, this invention provides an encrypted alternative implementation of a unique anchor point module. In this alternative, the model feature unique anchor point global constraint unit will use a high-dimensional random anchor point vector... The system replaces the traditional model-specific encrypted anchor matrix with one generated using national cryptographic algorithms. It invokes encryption algorithms such as SM4 and utilizes the model's preset seed key to generate an encrypted anchor matrix with high entropy. This encrypted anchor matrix remains encrypted throughout its distribution to the domain-differentiated coding unit, the dynamic feature evolution differentiation unit, and the feature-output dedicated decoupling mapping unit. Each unit performs real-time decryption via a built-in secure computation module when performing calculations involving anchor vectors. This encryption mechanism enhances the uncopyability and attack resistance of the model's feature identifiers, ensuring that even when deployed in a restricted environment, the model's underlying feature identifiers cannot be reverse-engineered or counterfeited by third parties.
[0059] The aforementioned adaptive alternatives share a unified knowledge-feature-output three-tier differentiation logic with the core implementation. By adjusting the calculation formulas of local modules, the applicability of this invention under different technical constraints is expanded. The result vectors generated by each alternative also follow a full-link anchor point correlation calibration mechanism, ensuring the consistency and long-term effectiveness of the large model's dehomogenization effect under different implementation scenarios.
Claims
1. A method for dehomogenizing large models based on quality diversity balancing and dynamic architecture, characterized in that: Includes the following steps: The multi-source heterogeneous knowledge input unit is used to preprocess the received multi-source heterogeneous data and generate knowledge type labels based on the data attributes. A normalized random high-dimensional vector is generated using the global constraint unit of the unique anchor point of the model feature, which serves as the unique anchor point vector of the model feature, and the unique anchor point vector of the model feature is embedded into the parameter initialization process of the subsequent processing unit. The domain-specific differential coding unit constructs a non-overlapping exclusive coding space based on the knowledge type label, and performs domain-specific differential coding on the preprocessed data in combination with the unique anchor vector of the model feature to generate a knowledge coding vector. The knowledge encoding vector is input into the base model, and the feature similarity change rate during the training process of the base model is calculated using the dynamic feature evolution differentiation unit to determine the feature evolution stage. Based on the feature evolution stage and the unique anchor vector of the model feature, the hidden layer feature vector of the base model is subjected to phased dynamic feature differentiation constraints. The constrained feature vector is input into the dedicated decoupling mapping network using a feature-output dedicated decoupling mapping unit, and combined with the output style unique encoding vector generated by the transformation of the model feature unique anchor vector to generate a differentiated output prediction vector.
2. The method for dehomogenizing large models based on quality diversity balancing and dynamic architecture according to claim 1, characterized in that, The preprocessing steps of the multi-source heterogeneous knowledge input unit for the received multi-source heterogeneous data include: For text data, a byte-pair encoding algorithm is used to perform word segmentation to generate a sequence of tokens; For knowledge graph data, entity linking and relation extraction are performed to generate triple sequences, which are then vectorized. For both numerical and structured data, normalization processing is performed to map the numerical values to a preset range and generate structured feature vectors.
3. The method for dehomogenizing large models based on quality diversity balancing and dynamic architecture according to claim 1, characterized in that, The steps for generating unique anchor vectors for model features using the global constraint unit for unique anchor points include: A high-dimensional random vector is generated based on a random number generator, wherein the dimension of the high-dimensional random vector is set according to the dimension of the hidden layer of the base model. By calculating the magnitude of the high-dimensional random vector and dividing each element of the vector by the magnitude, a unique anchor vector of model features with a magnitude of 1 is obtained.
4. The method for dehomogenizing large models based on quality diversity balancing and dynamic architecture according to claim 1, characterized in that, The steps for constructing non-overlapping dedicated coding spaces using the domain-specific differentiated coding units include: Obtain the total dimension of the embedding layer of the basic model, and divide the total dimension of the embedding layer into multiple subspaces according to the number of knowledge type tags; Each type of knowledge is assigned a non-overlapping subspace as its dedicated encoding space.
5. The method for dehomogenizing large models based on quality diversity balancing and dynamic architecture according to claim 1, characterized in that, The steps for generating knowledge encoding vectors using the domain-specific differential coding unit include: Various types of knowledge are encoded using a dedicated encoding function, wherein the dedicated encoding function obtains the dedicated encoding result by multiplying the initial embedding vector of the input data with the dedicated encoding weight matrix, and superimposing the bias term and the unique anchor vector of the model features weighted by the anchor fusion coefficient; The results of each specific encoding are merged into a unified knowledge encoding vector through a masking mechanism.
6. The method for dehomogenizing large models based on quality diversity balancing and dynamic architecture according to claim 1, characterized in that, The steps for determining the feature evolution stage in the dynamic feature evolution differentiation unit include: Calculate the cosine similarity difference between the hidden layer feature vectors of the current training round and the previous training round, and use it as the feature similarity change rate; If the rate of change of the feature similarity is greater than 0.05, it is determined to be in the rapid evolution stage; If the rate of change of the feature similarity is between 0.01 and 0.05, it is determined to be in a stable evolution stage; If the rate of change of the feature similarity is less than 0.01, it is determined to be in the convergence stage.
7. The method for dehomogenizing large models based on quality diversity balancing and dynamic architecture according to claim 6, characterized in that, The steps for the dynamic feature evolution differentiation unit to execute phased dynamic feature differentiation constraints include: The constraint coefficients are matched according to the determined characteristic evolution stage, wherein the constraint coefficients corresponding to the rapid evolution stage, the stable evolution stage, and the convergence stage increase sequentially. The difference between the hidden layer feature vector and the unique anchor vector of the model feature is weighted based on the constraint coefficient and then accumulated onto the original hidden layer feature vector. Calculate the feature differentiation degree, and automatically increase the constraint coefficient when the feature differentiation degree is lower than a set threshold.
8. The method for dehomogenizing large models based on quality diversity balancing and dynamic architecture according to claim 1, characterized in that, The steps of generating differential output prediction vectors by the feature-output dedicated decoupling mapping unit include: The constrained feature vectors are processed by a two-layer fully connected network with independent input parameters. A linear transformation is performed on the model feature unique anchor vector using the style encoding weight matrix to generate the output style unique encoding vector. The output vector of the two-layer fully connected network is added to the output style unique encoding vector to obtain the differentiated output prediction vector.
9. The method for dehomogenizing large models based on quality diversity balancing and dynamic architecture according to claim 1, characterized in that, It also includes end-to-end correlation calibration steps: After the processing steps of domain-specific differential encoding, phased dynamic feature differentiation constraints, and generation of differential output prediction vectors, the cosine similarity between the current result vector and the unique anchor vector of the model features is calculated. If the cosine similarity is less than 0.8, a linear transformation is performed on the resulting vector to improve its similarity to the unique anchor vector of the model features.
10. The method for dehomogenizing large models based on quality diversity balancing and dynamic architecture according to claim 1, characterized in that, It also includes steps to ensure the entire lifecycle of the model: During the inference phase after model training convergence and during multiple rounds of fine-tuning, the embedding state of the unique anchor vector of the model feature in the system architecture is maintained. During multiple rounds of fine-tuning, the value of the unique anchor vector of the model features is kept fixed and is not updated with the backpropagation algorithm.