A method and system for transferring a large model based on image representation of sparse activation

Abstract

The application discloses a kind of big model migration methods and systems based on image representation sparse activation, it is related to image processing and migration field, method includes: extracting image hidden representation, high importance expert is filtered from rank level expert set using gated network and sparse activation function, generate low rank adaptation parameter in scene;Second, different scene is assigned exclusive low rank adaptation subspace, orthogonal constraint is applied and orthogonal decoupling loss is constructed, realize multi-scene feature decoupling;Finally, with expert importance score as routing signal, cross-scene fusion weight is calculated through parameterless convergence mechanism, and unified adapter is generated by dynamically aggregating low rank parameter, and the big model is efficiently migrated to the current scene.The application realizes the efficient migration of pre-training big model under multi-scene image task by the technical means of task-in rank level expert sparse activation and task inter low rank subspace orthogonal decoupling.

Description

Technical Field

[0001] This invention relates to the field of image processing and transfer technology, specifically to a large model transfer method and system based on sparse activation of image representations. Background Technology

[0002] Large model fine-tuning methods have been widely used in various task transfers. While existing low-rank adaptation techniques can achieve parameter adaptation with relatively low parameter overhead, they are still prone to performance degradation during real-world scenario transfers. The main reason is that existing methods typically use fixed rank configurations, making it difficult to flexibly adapt to data differences within different scenarios, thus limiting the model's representational capabilities within those scenarios. Furthermore, when transferring to multiple scenarios, existing methods often lack effective separation mechanisms between different scenarios, easily leading to subspace overlap problems. This can confuse the decision boundaries of different scenarios, affecting the model's transfer performance during scenario switching and cross-scenario applications.

[0003] In recent years, with the widespread deployment of pre-trained large models in the field of image processing, efficient parameter fine-tuning methods have become a key approach to transferring large models to diverse downstream scenarios. Among them, low-rank adaptation technology, by introducing low-rank adjustable matrices into the pre-trained weight bypass, achieves task adaptation with a minimal number of parameters and has been widely used in various transfer scenarios. However, existing low-rank adaptation methods are still prone to performance degradation during real-world image scene transfer, and their limitations are mainly reflected in two aspects.

[0004] At the intra-scene adaptation level, existing methods typically employ a uniform and fixed rank configuration across the entire task space. This makes it difficult to flexibly adjust the rank based on the complexity and feature differences of the data within a scene, leading to insufficient representation capacity for important scenes or redundant parameters for simple scenes. This restricts the model's fitting ability and generalization performance within the scene. At the inter-scene transfer level, when the model needs to sequentially adapt to multiple significantly different image scenes, the low-rank adaptation parameters learned by different scenes are often located in unconstrained subspaces, lacking an effective orthogonal separation mechanism, which easily leads to high subspace overlap. This overlap not only confuses the decision boundaries of different scenes but also introduces cross-scene knowledge interference and catastrophic forgetting, severely reducing the model's transfer stability and accuracy in scene switching and cross-scene applications. Furthermore, existing methods also have significant shortcomings in the dynamic aggregation of multi-scene knowledge, making it difficult to adaptively fuse information from multiple scenes to form a unified and efficient adapter. Therefore, there is an urgent need for a new transfer paradigm that can adaptively configure rank capacity according to scene characteristics, achieve feature decoupling between scenes, and flexibly aggregate transfer gains from multiple scenes. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a large model transfer method and system based on sparse activation of image representations. By combining intra-task rank expert sparse activation with inter-task low-rank subspace orthogonal decoupling, efficient transfer of pre-trained large models under multi-scene image tasks is achieved.

[0006] On the one hand, a large model transfer method based on sparse activation of image representations includes:

[0007] S1. Acquire real-world scene image data and extract hidden representations from the real-world scene image data using a pre-trained visual encoder. Given a pre-trained large model, initialize a set of rank experts including multiple candidate rank values. Calculate the importance score for each rank expert in the rank expert set using a gating network, and select several rank experts with scores higher than the set importance score from the rank expert set using a sparse activation function, thus obtaining a subset of selected rank experts. Dynamically generate low-rank parameter increments based on the rank experts in the selected subset of rank experts to update the pre-trained large model, and output the low-rank adaptation parameters after sparse activation in the real-world scene image data and the importance score corresponding to each rank expert.

[0008] S2 takes the hidden representations and low-rank adaptation parameters of real scene image data as input, and assigns a dedicated low-rank adaptation subspace for each scene based on the distribution differences and semantic features of the hidden representations of real scene image data. Orthogonality constraints are applied to the dedicated low-rank adaptation subspaces of different scenes, and an orthogonal decoupling loss function is constructed to keep the parameter update directions induced by the image features of different scenes approximately orthogonal in the vector space, and outputs independent subspace parameters for decoupling image features of multiple scenes.

[0009] S3 uses the importance score corresponding to each rank expert as a global routing signal and the output independent subspace parameters as parameters to be aggregated. It calculates the fusion weight of each scene through a parameterless aggregation mechanism, dynamically aggregates the low-rank parameters in multiple scene-specific low-rank adaptation subspaces based on the fusion weight, generates a unified low-rank adapter that matches the current real scene image, and drives the updated pre-trained large model to achieve migration to the current scene through the unified low-rank adapter.

[0010] Specifically, in S1, a sparse activation function is used to select the top several rank experts from the rank expert set whose importance scores are higher than the set. The calculation formula is as follows:

[0011] ;

[0012] ;

[0013] ;

[0014] in, Represents a set of rank-based experts; rank-based experts Represents rank value ; The index number representing the hidden representation of the scene image data; Indicates importance score; Image data representing real-world scenes; pre-trained visual encoder ; Hidden representations of scene image data; Indicates the number of hidden representations; Indicates the preceding A rank-level expert; Indicates the preceding An index set of experts; This represents a sparse activation function.

[0015] Specifically, in S2, a rank-based sparse loss function was also designed. This is used to condense the knowledge within a scenario onto a critical minority of experts, ensuring the effective preservation of important parameters within the scenario, as follows:

[0016] ;

[0017] in, This indicates a rank-based expert, corresponding to a rank value. , The index number indicating the rank of the expert; The hidden representation of scene image data, Indicates the index number of the hidden representation; Indicates the gating weight; This represents the total number of rank values.

[0018] Specifically, in S2, the formula for calculating the orthogonal decoupling loss function is as follows:

[0019] ;

[0020] ;

[0021] in, Denotes the Frobenius inner product of matrices; and Representing different scenarios and A matrix in a low-rank subspace; This represents the projection function, used to determine its orthogonality; The square of the Frobenius norm of the matrix is ​​given by . This represents the orthogonal decoupling loss function; Representing a scene A low-rank increasing matrix; Representing a scene A low-rank increasing matrix; Representing a scene A low-rank descent matrix; Representing a scene A low-rank descent matrix.

[0022] Specifically, in S2, orthogonality constraints are applied to the low-rank fitting subspaces specific to different scenes, and an orthogonal decoupling loss function is constructed to ensure that the parameter update directions induced by image features of different scenes remain approximately orthogonal in the vector space. This includes:

[0023] S21. Obtain the low-rank fitness subspace matrix specific to the current scenario, and use the singular value decomposition algorithm to decompose the low-rank fitness subspace matrix into a left singular matrix, a singular value matrix, and a right singular matrix.

[0024] S22, based on the square root of the singular value in the singular value matrix, scales the singular vectors in the left and right singular matrices to moderate the parameter update magnitude, thereby dynamically updating the ascending and descending matrices in the low-rank fitting subspace.

[0025] S23, calculate the Frobenius inner product between the subspace matrices reconstructed by the updated ascending and descending matrices of any two different scenes, and determine their orthogonality by the projection function; if the inner product result is not zero, it is determined that there is an overlapping region in the subspaces of the two different scenes.

[0026] S24. Based on the discrimination results, an orthogonal decoupling loss function is constructed. By minimizing the orthogonal decoupling loss function, the dedicated low-rank adaptation subspaces of all scenes are driven to remain mutually orthogonal in the vector space, thereby ensuring that the image features of each scene can be independently modeled in its dedicated subspace and avoiding confusion of feature representations of different scenes.

[0027] Specifically, in S3, the fusion weights for each scene are calculated through a parameterless aggregation mechanism. Based on the fusion weights, low-rank parameters in multiple scene-specific low-rank adaptation subspaces are dynamically aggregated. The calculation formula is as follows:

[0028] ;

[0029] ;

[0030] in, It is a fixed, randomly initialized matrix. M represents a real number and is used to control the scale of R; This indicates a global average pooling operation; Indicates importance score; Representing a scene Matrix weights; Represented by natural constant An exponential function with base 0; Representing a scene The fusion weights; This represents a low-rank ascending matrix after cross-scene fusion; Representing a scene A low-rank increasing matrix; This represents the low-rank descent matrix after cross-scene fusion; Representing a scene A low-rank descent matrix.

[0031] On the other hand, a large model transfer system based on rank-level expert sparse activation includes:

[0032] The importance score calculation module is used to acquire real-world scene image data and extract hidden representations from the real-world scene image data through a pre-trained visual encoder. Given a pre-trained large model, a set of rank experts including multiple candidate rank values ​​is initialized. An importance score is calculated for each rank expert in the rank expert set through a gating network, and a sparse activation function is used to select the top several rank experts with higher importance scores from the rank expert set, resulting in a selected subset of rank experts. Based on the rank experts in the selected subset of rank experts, low-rank parameter increments are dynamically generated to update the pre-trained large model, and the low-rank adaptation parameters after sparse activation in the scene and the importance score corresponding to each rank expert are output.

[0033] The independent subspace parameter output module takes the hidden representations and low-rank adaptation parameters of real scene image data as input. Based on the distribution differences and semantic features of the hidden representations of real scene image data, it assigns a dedicated low-rank adaptation subspace for each scene. Orthogonality constraints are applied to the dedicated low-rank adaptation subspaces of different scenes, and an orthogonal decoupling loss function is constructed to ensure that the parameter update directions induced by the image features of different scenes remain approximately orthogonal in the vector space. The module outputs independent subspace parameters for decoupling image features across multiple scenes.

[0034] The migration module uses the importance score corresponding to each rank expert as a global routing signal and the output independent subspace parameters as parameters to be aggregated. It calculates the fusion weights of each scene through a parameterless aggregation mechanism, dynamically aggregates the low-rank parameters in multiple scene-specific low-rank adaptation subspaces based on the fusion weights, generates a unified low-rank adapter that matches the current real scene image, and drives the updated pre-trained large model to achieve migration to the current scene through the unified low-rank adapter.

[0035] The present invention adopts the above technical solution and has the following beneficial effects:

[0036] (1) This invention uses the subspace reparameterization and scaling update technology based on singular value decomposition to scale the singular vector by using the square root of the singular value, and dynamically updates the ascending and descending matrices, thereby effectively mitigating the parameter update amplitude within the scene, avoiding violent oscillations during the parameter update process, and ensuring the stability of low-rank adaptation parameter update in a single scene.

[0037] (2) This invention uses the Frobenius inner product calculation and orthogonal projection discrimination technique based on the reconstruction matrix to calculate the inner product of the subspace matrix reconstructed by the updated ascending matrix and descending matrix, constructs an orthogonal decoupling loss function, realizes the mutual orthogonality of the exclusive subspaces of different scenes in the vector space, ensures the independent modeling of the image features of each scene, and avoids the mutual confusion of the feature representations of multiple scenes.

[0038] (3) This invention achieves accurate condensation of knowledge within a scene and efficient separation of features between scenes by combining the collaborative optimization technology of sparse activation within a scene and orthogonal decoupling across scenes with gating network screening of key rank experts and orthogonal constraint mechanism of multi-scene subspaces. This significantly improves the accuracy and robustness of large model transfer to multiple scenes. Attached Figure Description

[0039] Figure 1 This is a flowchart of the large model transfer method based on sparse activation of image representation according to an embodiment of the present invention;

[0040] Figure 2 This is a framework diagram of the model transfer process in an embodiment of the present invention;

[0041] Figure 3 This is a diagram of a large model transfer system based on sparse activation of image representation, according to an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0043] like Figure 1 As shown, the present invention provides a large model transfer method based on sparse activation of image representations, comprising:

[0044] S1. Acquire real-world scene image data and extract hidden representations from the real-world scene image data using a pre-trained visual encoder. Given a pre-trained large model, initialize a set of rank experts including multiple candidate rank values. Calculate the importance score for each rank expert in the rank expert set using a gating network, and select several rank experts with scores higher than the set importance score from the rank expert set using a sparse activation function, thus obtaining a subset of selected rank experts. Dynamically generate low-rank parameter increments based on the rank experts in the selected subset of rank experts to update the pre-trained large model, and output the low-rank adaptation parameters after sparse activation in the scene and the importance score corresponding to each rank expert.

[0045] Specifically, in S1, a sparse activation function is used to select the top several rank experts from the rank expert set whose importance scores are higher than the set. The calculation formula is as follows:

[0046] ;

[0047] ;

[0048] ;

[0049] in, Represents a set of rank-based experts; rank-based experts Represents rank value ; The index number representing the hidden representation of the scene image data; Indicates importance score; Image data representing real-world scenes; pre-trained visual encoder ; Hidden representations of scene image data; Indicates the number of hidden representations; Indicates the preceding A rank-level expert; Indicates the preceding An index set of experts; This represents a sparse activation function.

[0050] Specifically, the optimal rank differs for different target layers within the scenario. Therefore, this design utilizes rank experts and importance assessment to achieve dynamic rank selection. Given a total number of ranks... Candidate rank set For each candidate rank Defined as a rank expert During the model training (fine-tuning) phase, forward parameter updates are performed to dynamically generate weight increments. Simultaneously, a rank-sparse loss function is designed. This involves condensing the knowledge within the scenario to the most critical few rank experts to ensure the effective preservation of important parameters within the scenario, as follows:

[0051] ;

[0052] in, Indicates a rank-based expert. Indicate its importance score, This represents the gating weights, which are mapped to scalar weights used for sparsity constraints.

[0053] S2 takes the hidden representations and low-rank adaptation parameters of real scene image data as input. Based on the distribution differences and semantic features of the hidden representations of real scene image data, it assigns a dedicated low-rank adaptation subspace for each scene. Orthogonality constraints are applied to the dedicated low-rank adaptation subspaces of different scenes. By constructing an orthogonal decoupling loss function, the parameter update directions induced by the image features of different scenes are kept approximately orthogonal in the vector space. The output is an independent subspace parameter for decoupling image features of multiple scenes.

[0054] Specifically, in S2, the rank-sparse loss function This is used to condense the knowledge within a scenario into a key minority of experts, ensuring the effective preservation of important parameters within the scenario:

[0055] ;

[0056] in, This indicates a rank-based expert, corresponding to a rank value. , The index number indicating the rank of the expert; The hidden representation of scene image data, Indicates the index number of the hidden representation; Indicates the gating weight; This represents the total number of rank values.

[0057] Specifically, in S2, the formula for calculating the orthogonal decoupling loss function is as follows:

[0058] ;

[0059] ;

[0060] in, Denotes the Frobenius inner product of matrices; and Representing different scenarios and A matrix in a low-rank subspace; This represents the projection function, used to determine its orthogonality; The square of the Frobenius norm of the matrix is ​​given by . This represents the orthogonal decoupling loss function; Representing a scene A low-rank increasing matrix; Representing a scene A low-rank increasing matrix; Representing a scene A low-rank descent matrix; Representing a scene A low-rank descent matrix.

[0061] Specifically, in S2, orthogonality constraints are applied to the low-rank fitting subspaces specific to different scenes. By constructing an orthogonal decoupling loss function, the parameter update directions induced by image features from different scenes are kept approximately orthogonal in the vector space. This includes:

[0062] S21. Obtain the low-rank fitness subspace matrix specific to the current scenario, and use the singular value decomposition algorithm to decompose the low-rank fitness subspace matrix into a left singular matrix, a singular value matrix, and a right singular matrix.

[0063] S22, based on the square root of the singular value in the singular value matrix, scales the singular vectors in the left and right singular matrices to moderate the parameter update magnitude, thereby dynamically updating the ascending and descending matrices in the low-rank fitting subspace.

[0064] S23, calculate the Frobenius inner product between the subspace matrices reconstructed by the updated ascending and descending matrices of any two different scenes, and determine their orthogonality by the projection function; if the inner product result is not zero, it is determined that there is an overlapping region in the subspaces of the two different scenes.

[0065] S24. Based on the discrimination results, an orthogonal decoupling loss function is constructed. By minimizing the orthogonal decoupling loss function, the dedicated low-rank adaptation subspaces of all scenes are driven to remain mutually orthogonal in the vector space, thereby ensuring that the image features of each scene can be independently modeled in its dedicated subspace and avoiding confusion of feature representations of different scenes.

[0066] Specifically, in this embodiment, different low-rank subspaces are assigned to each scene. Based on the orthogonality of parameters in the random projection process, the decision boundaries between scenes are effectively separated. The specific process is as follows:

[0067] Given a set of real-world scenarios For each scenario Allocate a dedicated low-rank subspace This subspace consists of a pair of scene-specific low-rank matrices. Zhang Cheng, used to generate weight increments unique to this scenario. First, regarding the first... In the scene Layers and subspaces are initialized to , The variance of the initial noise is represented; the subspace update process is as follows: The low-rank matrix is ​​decomposed into three matrices using Singular Value Decomposition (SVD). The subspace is then dynamically updated by scaling the singular vectors with the square roots of the corresponding singular values, as follows:

[0068] ;

[0069] in, and Let represent the ascending matrix and descending matrix in the low-rank subspace, respectively; and Let the left singular matrix and the right singular matrix represent the following respectively. Represents a singular value matrix; The square root operation is used to moderate the amplitude; finally, the decision boundaries between scenarios are effectively separated, and orthogonality constraints are imposed on the low-rank subspaces of different scenarios. This constraint is based on random projection theory: after random initialization, optimization is used to ensure that the low-rank matrices of different scenarios remain orthogonal in the projection space.

[0070] Specifically, a scene refers to a data domain divided based on the acquisition environment, visual distribution characteristics, and semantic category labels of real-world scene image data; each scene corresponds to a set of image samples that are similar in image distribution characteristics and model transfer requirements. Accordingly, in this paper, "each scene" refers to each data domain obtained from real-world scene image data according to the above-mentioned division rules.

[0071] S3 uses the importance score corresponding to each rank expert as a global routing signal and the output independent subspace parameters as parameters to be aggregated. It calculates the fusion weight of each scene through a parameterless aggregation mechanism, dynamically aggregates the low-rank parameters in multiple scene-specific low-rank adaptation subspaces based on the fusion weight, generates a unified low-rank adapter that matches the current real scene image, and drives the updated pre-trained large model to achieve migration to the current scene through the unified low-rank adapter.

[0072] Specifically, in S3, cross-scene fusion weights are calculated through a parameterless aggregation mechanism. Based on these fusion weights, low-rank parameters from multiple scene-specific low-rank adaptation subspaces are dynamically aggregated, including:

[0073] ;

[0074] ;

[0075] in, It is a fixed, randomly initialized matrix. M represents a real number and is used to control the scale of R; This indicates a global average pooling operation; Indicates importance score; Representing a scene Matrix weights; Represented by natural constant An exponential function with base 0; Representing a scene The fusion weights; This represents a low-rank ascending matrix after cross-scene fusion; Representing a scene A low-rank increasing matrix; This represents the low-rank descent matrix after cross-scene fusion; Representing a scene A low-rank descent matrix.

[0076] Specifically, the core of the implicit router designed in this embodiment lies in its importance score as described above. As a global routing signal, it is processed through a parameterless global average pooling. Parameter mapping yields cross-scene fusion weights This is used to guide model aggregation for multiple scenarios. The formula for model aggregation for multiple scenarios is as follows:

[0077] ;

[0078] ;

[0079] in, This represents the final large model; This represents a pre-trained large model; Let represent the low-rank increment matrix of the model. The overall optimization objective of the model is as follows:

[0080] ;

[0081] in, This represents the contrastive loss function, designed to maintain the generalizability of large models. This represents the weight hyperparameters for each loss term.

[0082] Specifically, such as Figure 2As shown, the left-hand module implements sparse activation within a task through rank expectation partitioning and a routing mechanism, and filters parameters using sparsity constraints. The right-hand module decouples parameter representations across tasks by using low-rank subspace allocation and orthogonality constraints to reduce conflicts. The bottom loss function integrates classification loss, sparse regularization, and orthogonality regularization, and calculates cross-task fusion weights using an implicit routing mechanism, thereby achieving efficient parameter utilization and effective decoupling of task heterogeneity in multi-task learning.

[0083] Specifically, some experimental configurations and partial experimental results of this invention are as follows:

[0084] Experimental configuration of this invention: The embodiments are planned to run on an Ubuntu 18.04.5 LTS system, equipped with an Intel Xeon E5-2698 v4 CPU, 128GB RAM, and a GeForce RTX 3090 GPU with 24GB of video memory. Model training is based on PyTorch 1.13 and CUDA 11.7, with NumPy used for data processing and Matplotlib for result visualization. Git is used for code version control. Training lasts for 120 epochs, with an initial learning rate of 3.5 × 10⁻⁴, decaying by a factor of 10 at the 40th and 90th epochs. The optimizer uses the Adam algorithm with a weight decay rate of 5 × 10⁻⁴, a momentum of 0.9, and a batch size of 64.

[0085] The experimental datasets and performance metrics of this invention are as follows: For image-text retrieval scenarios, the Flickr30K and MS-COCO datasets are used. Flickr30K contains approximately 31,000 images, each accompanied by 5 manually annotated descriptive texts, and is a classic benchmark for evaluating cross-modal retrieval performance. The MS-COCO dataset contains approximately 123,000 images, each also accompanied by 5 descriptive texts, and is widely used for image-text matching and image description scenarios. All datasets are divided into training and test sets according to the official protocol. For performance evaluation, standard metrics from the large model domain are used. For image-text retrieval scenarios, the bidirectional Recall@K (R@1, R@5, R@10) metric is used to measure the accuracy of text-to-image and image-to-text retrieval; for visual question answering scenarios, the Accuracy metric is used to evaluate the model's correct answer to visual questions; for image description scenarios, metrics such as CIDEr and BLEU-4 are used to comprehensively evaluate the quality of generated descriptions. In addition, frames per second (FPS) are used to measure the computational efficiency of the model in the migration scenario, so as to comprehensively evaluate the migration effect of this method in large model migration scenarios.

[0086] The key to this invention lies in designing a large-scale model transfer mechanism that combines rank-based expert sparse activation, subspace orthogonal decoupling, and implicit routing. This addresses the problems of existing low-rank adaptation methods in real-world scenario transfer, such as fixed rank configurations within a scenario, difficulty in flexible adaptation, and overlapping subspaces between scenarios, leading to confusion of decision boundaries between different scenarios. Specifically, this invention uses rank-based expert sparse activation to sparsely activate the low-rank adaptation branches corresponding to different candidate ranks, flexibly adapting to the rank distribution within a scenario; it uses subspace orthogonal decoupling to allocate independent low-rank subspaces for different scenarios and apply orthogonal constraints to effectively separate decision boundaries between scenarios; furthermore, it uses an implicit routing method with automatic scenario matching to automatically select the combined path of rank branches and subspaces based on different scenario data, improving the performance of the transferred model without introducing additional trainable parameters.

[0087] Therefore, this invention can be widely applied in fields such as intelligent video surveillance, pedestrian re-identification systems, and multi-scenario cross-domain model transfer.

[0088] like Figure 3 As shown, this embodiment also discloses a large model transfer system based on rank-level expert sparse activation, including:

[0089] Importance score calculation module 31 is used to acquire real scene image data and extract hidden representations of real scene image data through a pre-trained visual encoder; given a pre-trained large model, it initializes a set of rank experts including multiple candidate rank values; it calculates the importance score for each rank expert in the rank expert set through a gating network, and selects several rank experts with higher than the set importance score from the rank expert set using a sparse activation function to obtain the selected rank expert subset; it dynamically generates low-rank parameter increments based on the rank experts in the selected rank expert subset to update the pre-trained large model, and outputs the low-rank adaptation parameters after sparse activation in the scene and the importance score corresponding to each rank expert;

[0090] The independent subspace parameter output module 32 is used to take the hidden representation and low-rank adaptation parameters of real scene image data as input, and allocate a dedicated low-rank adaptation subspace for each scene based on the distribution differences and semantic features of the hidden representation of real scene image data. Orthogonality constraints are applied to the dedicated low-rank adaptation subspaces of different scenes, and by constructing an orthogonal decoupling loss function, the parameter update directions induced by the image features of different scenes are kept approximately orthogonal in the vector space, and the independent subspace parameters for decoupling image features of multiple scenes are output.

[0091] The migration module 33 is used to take the importance score corresponding to each rank expert as a global routing signal and the output independent subspace parameters as parameters to be aggregated. It calculates the fusion weight of each scene through a parameterless aggregation mechanism, dynamically aggregates the low-rank parameters in multiple scene-specific low-rank adaptation subspaces based on the fusion weight, generates a unified low-rank adapter that matches the current real scene image, and drives the updated pre-trained large model to achieve migration to the current scene through the unified low-rank adapter.

[0092] A specific implementation of a large model transfer system based on image representation sparse activation is described in this embodiment, which is the same as the large model transfer method based on image representation sparse activation.

[0093] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A large model transfer method based on sparse activation of image representations, characterized in that, Includes the following steps: S1. Acquire real-world scene image data and extract hidden representations from the real-world scene image data using a pre-trained visual encoder. Given a pre-trained large model, initialize a set of rank experts including multiple candidate rank values. Calculate the importance score for each rank expert in the rank expert set using a gating network, and select several rank experts with scores higher than the set importance score from the rank expert set using a sparse activation function, thus obtaining a subset of selected rank experts. Dynamically generate low-rank parameter increments based on the rank experts in the selected subset of rank experts to update the pre-trained large model, and output the low-rank adaptation parameters after sparse activation in the real-world scene image data and the importance score corresponding to each rank expert. S2 takes the hidden representations and low-rank adaptation parameters of real scene image data as input, and assigns a dedicated low-rank adaptation subspace for each scene based on the distribution differences and semantic features of the hidden representations of real scene image data. Orthogonality constraints are applied to the dedicated low-rank adaptation subspaces of different scenes, and an orthogonal decoupling loss function is constructed to keep the parameter update directions induced by the image features of different scenes approximately orthogonal in the vector space, and outputs independent subspace parameters for decoupling image features of multiple scenes. S3, take the importance score corresponding to each rank expert as the global routing signal, and take the output independent subspace parameters as the parameters to be aggregated; The fusion weights of each scene are calculated through a parameterless aggregation mechanism. Based on the fusion weights, the low-rank parameters in the low-rank adaptation subspaces of multiple scenes are dynamically aggregated to generate a unified low-rank adapter that matches the current real scene image. The unified low-rank adapter drives the updated pre-trained large model to achieve migration to the current scene.

2. The large model transfer method based on sparse activation of image representation according to claim 1, characterized in that, In S1, a sparse activation function is used to select the top few rank experts from the rank expert set whose scores are higher than the set importance score. The calculation formula is as follows: ； ； ； in, Represents a set of rank-based experts; rank-based experts Represents rank value ; The index number representing the hidden representation of the scene image data; The index number indicating the rank of the expert; Indicates importance score; Image data representing real-world scenes; This represents a pre-trained visual encoder; Hidden representations of scene image data; Indicates the number of hidden representations; Indicates the preceding A rank-level expert; Indicates the preceding A set of indexes for experts; the exp() exponential function; This represents a sparse activation function.

3. The large model transfer method based on sparse activation of image representation according to claim 1, characterized in that, In S2, a rank-based sparse loss function was also designed. This is used to condense the knowledge within a scenario onto a critical minority of experts, ensuring the effective preservation of important parameters within the scenario, as follows: ； in, This indicates a rank-based expert, corresponding to a rank value. , The index number indicating the rank of the expert; The hidden representation of scene image data, The index number representing the hidden representation of the scene image data; Indicates the gating weight; This represents the total number of rank values.

4. The large model transfer method based on sparse activation of image representation according to claim 1, characterized in that, In S2, the formula for calculating the orthogonal decoupling loss function is as follows: ； ； in, Denotes the Frobenius inner product of matrices; and Representing different scenarios and A matrix in a low-rank subspace; This represents the projection function, used to determine its orthogonality; The square of the Frobenius norm of the matrix is ​​given by . This represents the orthogonal decoupling loss function; Representing a scene A low-rank increasing matrix; Representing a scene A low-rank increasing matrix; Representing a scene A low-rank descent matrix; Representing a scene A low-rank descent matrix.

5. The large model transfer method based on sparse activation of image representation according to claim 1, characterized in that, In S2, orthogonality constraints are applied to the low-rank fitting subspaces specific to different scenes, and an orthogonal decoupling loss function is constructed to ensure that the parameter update directions induced by image features of different scenes remain approximately orthogonal in the vector space. Specifically, this includes: S21. Obtain the low-rank fitness subspace matrix specific to the current scenario, and use the singular value decomposition algorithm to decompose the low-rank fitness subspace matrix into a left singular matrix, a singular value matrix, and a right singular matrix. S22, based on the square root of the singular value in the singular value matrix, scales the singular vectors in the left and right singular matrices to moderate the parameter update magnitude, thereby dynamically updating the ascending and descending matrices in the low-rank fitting subspace. S23, calculate the Frobenius inner product between the subspace matrices reconstructed by the updated ascending and descending matrices of any two different scenes, and determine their orthogonality by the projection function; if the inner product result is not zero, it is determined that there is an overlapping region in the subspaces of the two different scenes. S24. Based on the discrimination results, an orthogonal decoupling loss function is constructed. By minimizing the orthogonal decoupling loss function, the dedicated low-rank adaptation subspaces of all scenes are driven to remain mutually orthogonal in the vector space, thereby ensuring that the image features of each scene can be independently modeled in its dedicated subspace and avoiding confusion of feature representations of different scenes.

6. The large model transfer method based on sparse activation of image representation according to claim 1, characterized in that, In S3, the fusion weights for each scene are calculated through a parameterless aggregation mechanism. Based on the fusion weights, low-rank parameters in multiple scene-specific low-rank adaptation subspaces are dynamically aggregated. The calculation formula is as follows: ； ； in, It is a fixed, randomly initialized matrix. M represents a real number and is used to control the scale of R; This indicates a global average pooling operation; Indicates importance score; Representing a scene Matrix weights; Represented by natural constant An exponential function with base 0; Representing a scene The fusion weights; This represents a low-rank ascending matrix after cross-scene fusion; Representing a scene A low-rank increasing matrix; This represents the low-rank descent matrix after cross-scene fusion; Representing a scene A low-rank descent matrix.

7. A large model transfer system based on rank-level expert sparse activation, characterized in that, include: The importance score calculation module is used to acquire real-world scene image data and extract hidden representations from the real-world scene image data through a pre-trained visual encoder. Given a pre-trained large model, initialize a set of rank experts including multiple candidate rank values; calculate the importance score for each rank expert in the rank expert set using a gating network, and select the top several rank experts with higher importance scores from the rank expert set using a sparse activation function to obtain the selected rank expert subset; dynamically generate low-rank parameter increments based on the rank experts in the selected rank expert subset to update the pre-trained large model, and output the low-rank adaptation parameters after sparse activation in the scene and the importance score corresponding to each rank expert; The independent subspace parameter output module takes the hidden representations and low-rank adaptation parameters of real scene image data as input. Based on the distribution differences and semantic features of the hidden representations of real scene image data, it assigns a dedicated low-rank adaptation subspace for each scene. Orthogonality constraints are applied to the dedicated low-rank adaptation subspaces of different scenes, and an orthogonal decoupling loss function is constructed to ensure that the parameter update directions induced by the image features of different scenes remain approximately orthogonal in the vector space. The module outputs independent subspace parameters for decoupling image features across multiple scenes. The migration module is used to take the importance score corresponding to each rank expert as a global routing signal and the output independent subspace parameters as parameters to be aggregated. The fusion weights of each scene are calculated through a parameterless aggregation mechanism. Based on the fusion weights, the low-rank parameters in the low-rank adaptation subspaces of multiple scenes are dynamically aggregated to generate a unified low-rank adapter that matches the current real scene image. The unified low-rank adapter drives the updated pre-trained large model to achieve migration to the current scene.

Images