Cross-domain cross-architecture visual base model knowledge fusion method and system

Abstract

The application provides a cross-domain cross-architecture visual base model knowledge fusion method and system. Student model and multiple heterogeneous teacher models are used to extract student and teacher visual features, the student and teacher visual features are uniformly represented as standard visual tokens, and the student standard visual tokens are projected to obtain student feature representations in M groups of teacher spaces. M learnable proxy tokens are introduced and combined with the student feature representations in the M groups of teacher spaces. An alternating dynamic interaction strategy is used to interact and fuse the combined features across spaces and within spaces. A multi-teacher global and local feature alignment strategy is used to align the final features after interaction and fusion with the teacher visual features to obtain visual global and local features for each teacher space, calculate feature loss, update the parameters of the student model and the projection layer parameters, and complete the cross-domain cross-architecture visual base model knowledge fusion of the student model.

Description

Technical Field

[0001] This invention relates to the field of visual model compression and visual universal representation learning technology. Specifically, it relates to a cross-domain and cross-architecture visual foundation model knowledge fusion method and system, and also to a corresponding image segmentation method, computer terminal, and computer-readable storage medium. Background Technology

[0002] Vision Foundation Models (VFMs), through large-scale self-supervised, reconstructive, visual-language alignment, or task-driven pre-training, can form general visual representations for tasks such as classification, segmentation, detection, depth estimation, and multimodal understanding. Different vision foundation models are significantly complementary. For example, models trained based on image-text alignment typically have strong global semantics and open vocabulary recognition capabilities; models trained based on mask reconstruction focus more on image structure and fine-grained texture; and models based on self-supervised representation learning can achieve strong object-level semantics and local consistency. How to integrate the capabilities of these various models into a unified model that is deployable, transferable, and adaptable to different student architectures is a key issue in visual model compression and general visual representation learning.

[0003] Existing multi-teacher knowledge distillation techniques primarily integrate the visual knowledge of multiple models into the student model by simultaneously aligning the visual features of the student model with those of multiple teacher models on a proxy dataset. Based on the method of visual representation alignment, existing methods can be broadly categorized into three types: 1. Direct feature alignment methods: The RADIO series of methods proposed by Heinrich et al. (Heinrich, G., Ranzinger, M., Yin, H., Lu, Y., Kautz, J., et al. Radiov2.5: Improved baselines for agglomerative vision foundation models. In CVPR (2025)) directly align the visual features of the student model with those of multiple base models such as SAM and CLIP globally and at the image patch level; 2. Introducing a carefully designed feature alignment module: The transformation from the visual features of the student model to the feature spaces of multiple teachers is often highly nonlinear, therefore, Sar... y ld z et al. (Sar) y ld z, MB, Weinzaepfel, P., Lucas, T., De Jorge, P., Larlus, et al. Dune: Distilling a universal encoder from heterogeneous 2d and 3d teachers. In CVPR (2025)) designed a complex Transformer module to align representations between students and multiple teachers, thereby integrating heterogeneous 2D and 3D knowledge into a single model; 3. Teacher weight adaptation: The model architecture and pre-training objectives of different teachers differ significantly, resulting in different statistical distributions of their visual features. Therefore, Wang et al. (Wang, Y., Azadani, MN, Sedwards, S. and Czarnecki, K. HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models. In NIPS (2025)) introduced a LoRA-based adaptive adjustment module for different teachers to maximize the transfer and integration of common knowledge among multiple teachers.

[0004] The existing solutions mentioned above have at least the following drawbacks in cross-domain, cross-architecture visual pedestal model knowledge fusion scenarios: 1. Representation space misalignment. CLIP, MAE, DINO, SAM, and various CNN / ViT pedestal models have different pre-training objectives, data sources, and feature geometric spaces. Direct alignment can easily cause student models to simultaneously receive mutually restrictive optimization signals, resulting in semantic conflicts and gradient interference. 2. Limited architecture transfer. CNN features usually have two-dimensional spatial priors, while ViT features are usually represented as one-dimensional token sequences. Traditional alignment methods often require designing dedicated projection structures for specific teacher-student combinations, making it difficult to reuse them uniformly among CNN teachers, ViT teachers, CNN students, and ViT students. 3. High data and computing costs. To alleviate conflicts among multiple teachers, some methods rely on billion-level proxy datasets or high-resolution multi-scale training, seeking representation consensus in a brute-force manner, resulting in excessively high training thresholds and computing costs. 4. Rigid knowledge fusion methods. Heinrich et al. (Heinrich, G., Ranzinger, M., Yin, H., Lu, Y., Kautz, J., et al. Radiov2.5: Improved baselines for agglomerative vision foundation models. In CVPR (2025)) directly aligned the student backbone network and multi-teacher outputs. However, they lacked an independent learning vehicle to first aggregate common knowledge among teachers and then feed the harmonized knowledge back to student features. Therefore, they are prone to transfer instability in fine-grained recognition, dense prediction, and resolution variation scenarios. Summary of the Invention

[0005] To address the aforementioned shortcomings in the existing technology, this invention provides a cross-domain, cross-architecture visual foundation model knowledge fusion method and system, and also provides a corresponding image segmentation method, computer terminal, and computer-readable storage medium.

[0006] According to a first aspect of the present invention, a cross-domain, cross-architecture visual foundation model knowledge fusion method is provided, comprising: Construct a heterogeneous set of teacher models and student models; The input image is fed into the student model and multiple teacher models of the heterogeneous teacher model set to obtain student visual features and teacher visual features, and all student visual features and teacher visual features are uniformly represented as a standard visual token. pass Each projection layer projects the student's standard visual token onto... The teacher standard visual token feature space corresponding to each teacher model is obtained. Student characteristic representation in a group teacher space; Introduction Each learnable proxy token, and associated with... The student feature representations in the group teacher space are combined to form Feature sequences in the group teacher space; An alternating dynamic interaction strategy is adopted for the above. The feature sequences under the group teacher space are interactively fused across space and within space; a multi-teacher global and local feature alignment strategy is adopted to align the final features after interactive fusion with the visual features of the teachers to obtain the visual global features and local features for each teacher space. Calculate the loss of the visual global features and local features for each teacher space, and update the parameters of the student model and the learnable parameters of the projection layer; Iterate through the above process until the termination condition is met, thus completing the cross-domain, cross-architecture visual foundation model knowledge fusion of the student model.

[0007] According to a second aspect of the present invention, a cross-domain, cross-architecture visual foundation model knowledge fusion system is provided, comprising: The multi-model distillation framework module is used to construct a heterogeneous set of teacher models and student models. The feature unification module is used to feed the input image into the student model and multiple teacher models of the heterogeneous teacher model set to obtain student visual features and teacher visual features, and to unify the student visual features and teacher visual features into a standard visual token. The teacher represents the space construction module, which is used to... Each projection layer projects the student's standard visual token onto... The teacher standard visual token feature space corresponding to each teacher model is obtained. Student characteristic representation in the group teacher space; introduction Each learnable proxy token, and associated with... The student feature representations in the group teacher space are combined to form Feature sequences in the group teacher space; The feature fusion interaction module employs an alternating dynamic interaction strategy for the features fused together. The feature sequences under the group teacher space are interactively fused across space and within space; a multi-teacher global and local feature alignment strategy is adopted to align the final features after interactive fusion with the visual features of the teachers to obtain the visual global features and local features for each teacher space. The model optimization module is used to calculate the loss of visual global features and local features for each teacher space, update the parameters of the student model and the learnable parameters of the projection layer, and complete the cross-domain and cross-architecture visual base model knowledge fusion of the student model.

[0008] According to a third aspect of the present invention, an image segmentation method is provided, comprising: Obtain an image segmentation visual base model that integrates cross-domain and cross-architecture knowledge, trained using the method described above in this invention; Acquire the target image data to be processed; The target image data is input into the image segmentation visual base model, and the image segmentation result is output through model inference to achieve the segmentation of the target image.

[0009] According to a fourth aspect of the present invention, a computer terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, can be used to execute the cross-domain, cross-architecture visual base model knowledge fusion method or image segmentation method described above in the present invention.

[0010] According to a fifth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the cross-domain, cross-architecture visual pedestal model knowledge fusion method or image segmentation method described above in the present invention.

[0011] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art: Existing multi-teacher distillation methods typically align student models to the feature spaces of multiple teacher models simultaneously. However, due to differences in pre-training objectives, network structures, and training data, the visual representations of different teacher models often exhibit significant variations, easily leading to competition for supervisory signals during the distillation process. The cross-domain, cross-architecture visual pedestal model knowledge fusion method and system provided in this invention decouples teacher representation spaces, establishing corresponding feature learning spaces for different teacher models. This allows student models to absorb knowledge in multiple teacher spaces, rather than being forced to match multiple heterogeneous teachers in a single space, thereby significantly reducing optimization interference caused by representation inconsistencies.

[0012] The cross-domain, cross-architecture visual base model knowledge fusion method and system provided by this invention introduces three quantitative indicators to evaluate teacher conflict in multi-teacher distillation, providing quantitative evaluation criteria for distillation hyperparameters.

[0013] The cross-domain, cross-architecture visual base model knowledge fusion method and system provided by this invention unifies all visual features of students and teachers into a one-dimensional token sequence. Transformer blocks can be used to more efficiently fuse information across teacher feature spaces for different teacher architectures. Unifying visual features into a token sequence also makes it more flexible to handle changes in image input resolution.

[0014] The cross-domain, cross-architecture visual foundation model knowledge fusion method and system provided by this invention can achieve or surpass existing multi-teacher distillation techniques using only the ImageNet-1K dataset of millions of samples. Compared to existing benchmark methods that rely on proxy datasets of billions of samples, this invention significantly reduces the requirements for training data scale and computing resources while maintaining high performance, achieving a significant improvement in data efficiency.

[0015] The cross-domain, cross-architecture visual foundation model knowledge fusion method and system provided by this invention possesses excellent architectural flexibility and versatility. It achieves architecture independence through a tokenization mechanism, supporting CNN student models including ResNet and ConvNeXt, as well as various ViT student models. Furthermore, without altering the backbone network of the student model, this invention, through the guidance of neural agents, enables the student model to capture richer semantic information and fine-grained features than a single teacher model, and exhibits strong robustness to changes in input image resolution.

[0016] The image segmentation method provided by this invention is based on a cross-domain, cross-architecture visual foundation model knowledge fusion method and system. In the downstream task of image segmentation, it achieves a comprehensive technical effect of high precision, high speed and high robustness, which emphasizes both details and the global picture, lightweight reasoning and multi-source knowledge fusion. It is especially suitable for applications in fields with extremely high requirements for safety and real-time performance, such as medical diagnosis, autonomous driving and industrial quality inspection. Attached Figure Description

[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a diagram illustrating the overall working architecture of the cross-domain, cross-architecture visual base model knowledge fusion method in a preferred embodiment of the present invention.

[0018] Figure 2 This is a diagram showing the semantic results of visual features extracted in a verification example of the present invention.

[0019] Figure 3This is a comparison chart of the optimization gradient differences of the multi-teacher model's contribution to student parameters in another verification example of the present invention. The higher the difference value, the more serious the teacher optimization conflict. Among them, (a) is the gradient difference value calculated using gradient cosine average similarity PCD, (b) is the gradient difference value calculated using gradient interference ratio GIR, and (c) is the gradient difference value calculated using gradient sign average difference SCR.

[0020] Figure 4 This is a scaling experiment of the model under different data scales in other verification examples of the present invention.

[0021] Figure 5 These are experimental results of the model on a multimodal dataset in other verification examples of this invention. Detailed Implementation

[0022] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0023] Existing multi-teacher visual pedestal model knowledge fusion technology has the following technical problems: different teacher models often come from different pre-training tasks, different network architectures and different data distributions. If the student model is directly aligned to multiple teacher models at the same time, it is easy to cause competition between teacher supervision signals, making it difficult for the student model to determine which visual knowledge to learn first during the optimization process.

[0024] To address the aforementioned issues, one embodiment of the present invention provides a cross-domain, cross-architecture visual pedestal model knowledge fusion method based on teacher representation space decoupling and conflict suppression. This method does not simply merge and align the features of multiple teacher models. Instead, it first constructs independent but interactive representation spaces for different teachers, and then achieves stable transfer of multi-source visual knowledge to a single student model through cross-space shared knowledge modeling, preservation of proprietary knowledge within the space, and conflict-aware distillation supervision. In this embodiment, "cross-domain" refers to the different types of training data and capabilities of the pedestal models, such as multimodal perception, macroscopic visual target perception, and fine-grained visual perception; "cross-architecture" refers to the knowledge transfer distillation process, which can range from convolution to Transformer.

[0025] Specifically, such as Figure 1 As shown, the cross-domain, cross-architecture visual foundation model knowledge fusion method provided in this embodiment may include: S1, Construct a heterogeneous set of teacher models and student models; S2, the input image is fed into the student model and multiple teacher models of the heterogeneous teacher model set respectively to obtain the student visual features and teacher visual features, and all student visual features and teacher visual features are uniformly represented as a standard visual token; S3, via Each projection layer projects the student's standard visual token onto... The teacher standard visual token feature space corresponding to each teacher model is obtained. Student characteristic representation in a group teacher space; S4, Introduction Each learnable proxy token, and associated with... The student feature representations in the group teacher space are combined to form Feature sequences in the group teacher space; S5 employs an alternating dynamic interaction strategy for... The feature sequences under the group teacher space are interactively fused across space and within space; a multi-teacher global and local feature alignment strategy is adopted to align the final features after interactive fusion with the teacher's visual features to obtain the visual global and local features for each teacher space. S6, calculate the loss of visual global and local features for each teacher space, and update the parameters of the student model and the learnable parameters of the projection layer. S7. Iterate the above process until the termination condition is met, and complete the cross-domain and cross-architecture visual base model knowledge fusion of the student model.

[0026] In some preferred embodiments, S1 above, constructing a heterogeneous set of teacher models and student models, may further include: S11, given multiple pre-trained visual pedestal models as a set of teacher models, denoted as... The teacher model may include, but is not limited to: image-text alignment model, mask reconstruction model, self-supervised vision model, classification pre-trained model, detection or segmentation pre-trained model, etc., and its network architecture may be CNN, ViT or CNN-Transformer hybrid architecture. S12, Construct a student model to be trained. The student model can employ CNN, ViT, or other visual encoder structures.

[0027] During training, the teacher model parameters remain frozen, and only the student model and its attached lightweight mapping function are updated.

[0028] In some preferred embodiments, S2 above, which involves feeding the input image into multiple teacher models of a student model and a heterogeneous teacher model set respectively, to obtain student visual features and teacher visual features, and uniformly representing all student visual features and teacher visual features as a standard visual token, may further include: For the input image The data were fed into student models and multiple teacher models respectively to obtain student visual features and teacher visual features. ; Based on the visual characteristics of students and teachers, the visual features generated by different architectures are uniformly organized into a standard visual unit representation, resulting in a standard visual token sequence. For CNN features, which have a distinct 2D spatial structure, their spatial locations (spatial structure) are unfolded into a sequence of one-dimensional spatial visual units to form visual tokens. For ViT features, which are already one-dimensional image token sequences, no further processing is required; their patch representations (image patch features) or category representations (category features) are directly used as sequence visual units. Finally, the visual features of the two architectures are unified into an image token sequence to obtain standard visual tokens.

[0029] This step is used to unify the visual representations for students and teachers. Since CNN outputs are typically two-dimensional spatial feature maps, while ViT outputs are typically one-dimensional sequence features, this step transforms the outputs of CNN and ViT into a unified visual unit description format, thereby supporting knowledge transfer between teachers and students across architectures.

[0030] In some preferred embodiments, the above-mentioned S3 is achieved by... Each projection layer projects the student's standard visual token onto... The teacher standard visual token feature space corresponding to each teacher model is obtained. The representation of student characteristics in a group teacher space can further include: Apply the standard visual token to each student A lightweight mapping function ,get Student characteristics in the group teacher space are represented as follows: in, The student model is represented in the first... The projection results in the teacher standard visual token feature space (teacher space) corresponding to each teacher model; Represents a standard visual token; Indicates the first A lightweight mapping function, the specific structure of which includes a fully connected-GELU nonlinear-fully connected layer.

[0031] Each mapping function The student features, i.e., the standard visual tokens of students, are transformed into a representation distribution that is closer to the corresponding teacher model, thus obtaining feature representations that are adapted to each teacher space. Through this representation space decoupling strategy, the student model does not need to satisfy multiple conflicting teacher supervisions in the same feature space at the same time, thereby reducing optimization interference in multi-teacher distillation.

[0032] This step is used to construct a teacher-specific representation space. Since different teacher models exhibit significant differences in feature distribution, semantic preferences, and feature dimensions, this invention does not directly require student features to match all teacher features simultaneously. Instead, it constructs a corresponding teacher representation space for each teacher model.

[0033] In some preferred embodiments, the above-described S4 introduces... Each learnable proxy token, and associated with... The student feature representations in the group teacher space are combined to form The feature sequences in the group teacher space can further include: Will Each learnable proxy token is concatenated with the student feature representation under the corresponding teacher space on the channel to obtain... Feature sequences in the group teacher space.

[0034] In some preferred embodiments, the above-mentioned S5 employs an alternating dynamic interaction strategy for... The feature sequences under the group teacher space can be further integrated across and within the space through interactive fusion, and may include: The alternating dynamic interaction strategy includes alternating cross-space interaction mechanisms and intra-space interaction mechanisms; among them, the cross-space interaction mechanism is used to interact and integrate shared knowledge between different teacher spaces, and the intra-space interaction mechanism is used to interact and integrate specific knowledge within each teacher space. pass The alternation of cross-space interaction mechanisms and intra-space interaction mechanisms at each stage outputs the final characteristics after interaction fusion.

[0035] In some preferred embodiments, S5 above employs a multi-teacher global and local feature alignment strategy to align the final features after interactive fusion with the teacher's visual features, obtaining visual global and local features for each teacher's space. It may further include: The features output by the last spatial interaction mechanism are segmented by channel to obtain visual token features and learnable token features. The learnable token features are then used to obtain a global token representation through global average pooling. Aligning the global token representation with the global features of the teacher models, and dynamically fusing global visual knowledge from different teacher spaces by constraining learnable token features; specifically, for the Transformer model, the global feature of the teacher model is the cls token; for the convolutional model, the global feature of the teacher model is the result of global average pooling of teacher space features. Dynamically integrating global visual knowledge among teachers refers to dynamically aligning it with the global features of M groups of teacher models.

[0036] By using global feature alignment loss constraints, token features can be learned, dynamically integrating the visual features of multiple teacher models. Then, the fused features obtained through an alternating dynamic interaction strategy are used to optimize the visual token features of the student model in different teacher spaces, achieving local feature alignment with different teacher models. The basic components of the alternating dynamic interaction part are public space interaction (i.e., cross-space interaction mechanism) and private space interaction (i.e., intra-space interaction mechanism), which are stacked alternately. Depending on the student architecture, either convolutional or Transformer architecture is selected. Public space interaction concatenates the student feature channels of M teacher spaces to achieve feature fusion in the teacher spaces; private space interaction independently reorganizes features in the M teacher spaces, specializing the knowledge extracted from the public space to the corresponding teacher feature space.

[0037] The final output includes visual global and local features for each teacher's space.

[0038] This step provides a multi-teacher global and fine-grained feature alignment strategy. After obtaining student features in multiple teacher representation spaces, this invention fuses multi-teacher knowledge through a hierarchical interaction module. This process includes two parts: cross-space interaction and intra-space interaction. After several stages of cross-space and intra-space interactive processing, this invention guides the student model to optimize through a two-level feature alignment of global and local features, ensuring that it can master both macro-level semantics and cross-modal knowledge, while also inheriting fine-grained detailed features. Specifically, the feature output results of the last intra-space module are first processed... Segmented by channel, the visual token portion is obtained. With the learnable token part The Token section is available for learning. The global token representation is obtained through global average pooling. Subsequently, the global features of the teacher model and the final output of the learnable token part, namely the global token representation, are combined. Alignment is performed, and global visual knowledge between different teacher spaces is dynamically fused through constraints on the learned token component. This process can be represented as follows: Global representation alignment: ; in, It is a linear layer used to align the feature dimensions for calculating the loss; Indicates the first Global feature alignment loss for a single teacher model; Denotes the Frobenius norm; Indicates the first The global visual features of a teacher model. If it is a convolutional model, they are obtained by global average pooling of 2D features; if it is a Transformer model, they are obtained from the cls token.

[0039] Generally, different visual models show relatively small differences in global image representation, therefore... The learnable token portion can be constrained to dynamically integrate the visual features of multiple teacher models, and then the hierarchical interaction module can be used to optimize the visual token features of the student model in different teacher spaces, thereby achieving robust local feature alignment with different teacher models. This process is represented as follows: Local representation alignment: in, To indicate the first Local feature alignment loss of a teacher model; It is a linear layer used to align the feature dimensions for calculating the loss; Indicates average pooling; Represents the teacher model architecture; This indicates the teacher's visual characteristics.

[0040] Furthermore, for the CLIP model, its output features are only global multimodal visual features, therefore a linear layer is used first. Visual Token Features Dimensions and CLIP features Alignment, then average pooling The features are transformed into global features and then constrained using cosine similarity loss to ensure that CLIP's multimodal semantic knowledge is transferred to the student model. For the DINO self-supervised pre-trained model, a linear layer is first used. Feature dimension alignment is performed, and then the local semantic information of DINO is transferred to the student model using image patch-level cosine similarity loss; for other pre-trained models such as MAE, knowledge transfer is performed by L2 loss according to the reconstruction method.

[0041] To quantitatively evaluate optimization conflicts under multi-teacher distillation and compare the effectiveness of different algorithms in mitigating gradient conflicts in student model parameter updates, this invention introduces three quantitative indicators. Generally, let... For the first The gradient of the teacher model feature alignment loss with respect to the student model parameters. Here, the alignment loss is the total feature alignment loss of each teacher model with respect to the student features, which is generally the sum of the global feature alignment loss and the local feature alignment loss. Specifically, for the CLIP model, which has no local features, the alignment loss is the global feature alignment loss. Therefore: 1. Gradient Cosine Mean Similarity (PCD) 2. Gradient Interference Ratio (GIR) 3. Gradient Sign Mean Difference (SCR) in, Expressing expectations, This is used to ensure that duplicate terms are removed when calculating the sum and average. This indicates the calculation of cosine similarity between vectors. For sign functions. These three metrics can measure the degree of conflict between the alignment of features of different teacher models and the parameters of the student model during model training.

[0042] The specific applications of the quantitative evaluation results in the model training process are as follows: First, use metrics other than loss to monitor the model's learning behavior from a single teacher and determine whether the student model is learning from only one teacher; second, these metrics can be used to determine whether the feature dimensions of the learnable tokens are saturated. Learnable tokens introduce additional parameters to the alternation modules, and a balance needs to be achieved between feature dimensions and conflict mitigation. This balance can be determined using the quantitative metrics here; third, it provides a reference for the selection of learning rate and the number of alternation modules.

[0043] Based on the same inventive concept, another embodiment of the present invention provides a cross-domain, cross-architecture visual foundation model knowledge fusion system, which may include: The multi-model distillation framework module is used to construct a heterogeneous set of teacher models and student models. The feature unification module is used to feed the input image into the student model and multiple teacher models of the heterogeneous teacher model set to obtain the student visual features and the teacher visual features, and to unify the student visual features and the teacher visual features into a standard visual token. The teacher represents the space construction module, which is used to... Each projection layer projects the student's standard visual token onto... The teacher standard visual token feature space corresponding to each teacher model is obtained. Student characteristic representation in the group teacher space; introduction Each learnable proxy token, and associated with... The student feature representations in the group teacher space are combined to form Feature sequences in the group teacher space; The feature fusion interaction module employs an alternating dynamic interaction strategy for... The feature sequences under the group teacher space are interactively fused across space and within space; a multi-teacher global and local feature alignment strategy is adopted to align the final features after interactive fusion with the teacher's visual features to obtain the visual global and local features for each teacher space. The model optimization module is used to calculate the loss of visual global and local features for each teacher space, update the parameters of the student model and the learnable parameters of the projection layer, and complete the cross-domain and cross-architecture visual base model knowledge fusion of the student model.

[0044] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system. That is, the embodiments in the method can be understood as preferred examples for building the system, and will not be elaborated here.

[0045] Based on the same inventive concept, other embodiments of the present invention also provide an image segmentation method that can be used in downstream tasks such as industrial manufacturing and inspection, medical imaging and assisted diagnosis, autonomous driving and traffic monitoring, and agricultural ecology and environmental monitoring.

[0046] Specifically, the image segmentation method may include: M1, obtain the image segmentation visual base model that integrates cross-domain and cross-architecture knowledge, trained using the method provided in the above embodiments of the present invention; M2, Acquiring the target image data to be processed: In practical applications, the target image is optical image data containing a specific physical scene or object. Specifically, the target image is determined based on the specific downstream task. For example, the target image may include, but is not limited to, the following types: When image segmentation methods are applied to industrial manufacturing and inspection, the target image can be an image of surface defects in industrial products, an image of precision parts dimensions, an image of material sorting on an automated production line, etc. When image segmentation methods are applied to medical imaging and auxiliary diagnosis, the target images can be medical images such as X-rays, CT, MRI, and ultrasound, as well as images taken by the body surface / endoscope such as skin lesions and fundus retina; When image segmentation methods are applied to autonomous driving and traffic monitoring, the target image can be road scene images captured by vehicle cameras, traffic sign and marking images, vehicle and pedestrian target detection images, etc. When image segmentation methods are applied to agricultural ecology and environmental monitoring, the target images can be crop pest and disease detection images, plant growth status assessment images, wild animal identification and ecological monitoring images, etc. It should be noted that the types of target images described above are merely illustrative and are not intended to limit the scope of protection of this invention. Those skilled in the art will understand that any physical application scenario requiring the alignment, classification, or recognition of visual image data with textual semantic information falls within the scope of the image segmentation method of this invention.

[0047] M3 takes the target image data as input to the image segmentation visual base model, and outputs the image segmentation result through model inference to achieve the segmentation of the target image.

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, a specific application example is provided below to further illustrate the invention in detail. It should be understood that the specific application example described herein is merely illustrative and not intended to limit the invention. Modifications or equivalent substitutions made by those skilled in the art based on their understanding of the technical solutions of this invention, without departing from the spirit and scope of the invention, should be covered within the protection scope of this invention.

[0049] The cross-domain, cross-architecture visual foundation model knowledge fusion method provided by this invention is implemented as follows: Step 1: Construct a heterogeneous set of teacher and student models, which includes: Obtain the initial CNN and ViT vision pre-trained models. Specifically, this includes models (ResNet-50, ConvNeXt-B, ViT-B, and ViT-L) trained using ImageNet classification pre-training, DINO self-supervised pre-training, and MAE mask reconstruction self-training. Among them, ConvNeXt-B, ViT-B, and ViT-L use the latest open-source pre-trained models of DINOv3.

[0050] The proposed learnable token and the hierarchical interaction module for providing alternating dynamic interaction strategies are connected to the student model. The learnable agent token and the student feature representation in the teacher space are concatenated on the channel, and after passing through the alternating dynamic interaction strategy, the visual global and local features for each teacher space are output.

[0051] Step two involves acquiring the distillation proxy dataset and constructing the downstream performance evaluation dataset. This invention uses 1.26 million images from ImageNet-1k as training data, with the training images uniformly cropped to... Resolution. The downstream performance benchmark dataset uses the ImageNet-1k validation set, three fine-grained classification validation sets, the ADE20K validation set, and the NYUv2 test set.

[0052] Step 3: Minimize the loss function defined in this invention using the AdamW optimizer on the training dataset. (including global feature loss) and local feature loss To update the student model and alternating module parameters: During training, for the convolutional student model, the learnable token dimension was set to 1024 (ResNet-50) and 512 (ConvNeXt-B), respectively, and the number of hierarchical interaction modules was set to... For the Transformer student model, the learnable token dimensions are set to 512 (ViT-B) and 768 (ViT-L), respectively. The number of hierarchical interaction modules is set to... The entire training process follows the standard ImageNet-1k 300-epoch procedure, with an initial learning rate of 1e-4, weight decay of 0.01, and mixed precision training to reduce computational load.

[0053] Step four involves transferring the distilled student model to ImageNet-1k, FGVC, ADE20K, and NYUv2 classification, fine-grained classification, segmentation, and depth estimation tasks. On the ImageNet-1k training set, the student model and the first 10 epochs of classification are fine-tuned, and then the Top-1 accuracy is calculated. In the fine-grained classification FGVC task, this invention uses the Aircraft, Caltech101, and CUB datasets to fine-tune the student model, with the image resolution set to... The model was trained for 100 epochs and evaluated using Top-1 accuracy. On the ADE20K training set, the student model parameters were frozen, and the image resolution was set to... The segmentation head is trained using linear probing. This invention employs a standard 80k-iteration training process with a batch size of 16. After training, the mean Intersection over Union (mIoU) metric is used to evaluate performance. On the NYUv2 training set, this invention uses dense prediction Transformer DPT as the depth image decoder, concatenating the student model with the decoder in a multi-scale manner, and using a standard 25k-iteration training process. After training, the root mean square error (RMSE) is used for evaluation.

[0054] The technical effects achievable by the technical solutions provided in the above embodiments of the present invention will be further explained below with reference to specific verification examples. Verification Example 1: The qualitative verification of the representation quality of the method of the present invention on natural images, remote sensing images, and autonomous driving scenarios is conducted. Specifically, PCA technology is used to project visual features into a 3D RGB space for visualization.

[0055] Experimental results are as follows Figure 2 As shown, the method of this invention can extract richer semantic details compared to the pre-trained visual pedestal model MAE and DINOv3; compared to the concurrent distillation methods Proteus and Radiov2.5, the method of this invention extracts richer semantic details and more continuous semantic information within the image. These results indicate that the method of this invention has strong generalizability for general representation extraction in various scenarios.

[0056] Verification Example 2: The ViT model was selected as the teacher, and the method of the present invention was verified by distillation on the 1% ImageNet-1k dataset to effectively suppress the gradient direction differences between different teachers throughout the training process.

[0057] This invention uses gradient cosine mean similarity (PCD), gradient interference ratio (GIR), and gradient sign mean difference (SCR) to evaluate the gradient difference value of the teacher model in the training process. Experimental results are as follows: Figure 3 As shown in (a) to (c), DirectAlign refers to aligning student and teacher features directly using linear transformation and nonlinear activation without employing the method of this invention, while ACTok refers to the method of this invention. The method of this invention effectively suppresses gradient direction differences between different teachers throughout the entire training process.

[0058] Verification Example 3: The ViT model was selected as the teacher model, and the transferability of the proposed method was verified on downstream tasks such as classification, fine-grained classification, segmentation, and depth estimation.

[0059] Table 1. Quantitative comparison with existing methods on downstream tasks (teacher model using ViT architecture). The experimental results are shown in Table 1. The amount of surrogate data required by this invention is only 1.2M, approximately 0.1% of that required by DINOv3 and RADIOv2.5, indicating that the method of this invention has high data efficiency. Furthermore, compared to other methods that require larger ViT-G and ViT-H as teacher models, the teacher model ViT-B used in this invention is more lightweight. On ImageNet-1k classification and fine-grained classification tasks, the method of this invention outperforms current state-of-the-art distillation methods. Moreover, on dense segmentation and depth estimation tasks, the method of this invention performs comparably to current state-of-the-art methods. These results validate the training efficiency and transfer effectiveness of the method of this invention.

[0060] Verification Example 4: ResNet-50 and ConvNeXt-B convolutional models were selected as teacher models to verify the transfer effectiveness of the proposed method on downstream tasks such as classification, fine-grained classification, segmentation, and depth estimation.

[0061] Table 2 shows a quantitative comparison with existing methods on downstream tasks (teacher model using convolutional architecture). The experimental results are shown in Table 2. When using the ResNet-50 model, the proposed method achieves the best performance in fine-grained classification, segmentation, and depth estimation. When using the ConvNeXt-B model, the proposed method requires only about 0.1% of the data compared to DINOv3, and its performance on multiple tasks rivals current state-of-the-art methods. Furthermore, when the teacher model is set to ViT and the student model is set to ConvNeXt-B, the proposed method can still effectively achieve cross-architecture knowledge fusion. These results demonstrate that the proposed method can achieve multi-model knowledge fusion in cross-architecture, low-data-volume scenarios and can be efficiently transferred to various downstream vision tasks, showing promising application prospects and value.

[0062] Verification Example 5: The performance of our proposed method (ACTok) on multimodal datasets was compared using the NVIDIA E-RADIO-L model. Both methods employed the ViT-L student model as the visual encoder, utilized the standard LLaVA framework with Vicuna-7B-1.5 as the LLM base, trained cross-modal projection layers and performed instruction fine-tuning, and were then evaluated on the POPE and VQAv2 datasets. The results are as follows: Figure 4 As shown, the method of the present invention achieves 87% accuracy on POPE, which is 1.9% higher than E-RADIO-L, and requires less training data for the student model.

[0063] Verification Example 6: ViT-B was selected as the student model architecture. The student model was trained using 1%, 20%, 50%, and 100% of the data, and evaluated on three datasets (Caltech101, CUB, and Aircraft) to study the model's scaling performance. Figure 5 As shown, using more data can continuously improve the model's performance, demonstrating a good scaling effect.

[0064] The results from the above verification examples demonstrate that, on fine-grained image classification datasets, semantic segmentation, and depth estimation, the method of this invention surpasses teacher models trained on similar data scales, and its capabilities are comparable to, and even exceed, models trained on much larger datasets (approximately 1000 times the data size). It can be applied in downstream resource-constrained fields requiring fine-grained visual perception and multimodal perception. Furthermore, feature visualization results also indicate that the algorithm possesses finer-grained perception capabilities in autonomous driving scenarios and remote sensing.

[0065] This invention decouples teacher representation spaces, establishing corresponding feature learning spaces for different teacher models. This allows student models to absorb knowledge in multiple teacher spaces, rather than being forced to match multiple heterogeneous teachers in a single space, thus significantly reducing optimization interference caused by representation inconsistencies. Utilizing only the million-level ImageNet-1K dataset, it achieves or surpasses existing multi-teacher distillation techniques, significantly reducing the requirements for training data scale and computational resources while maintaining high performance, achieving a significant improvement in data efficiency. It possesses excellent architectural flexibility and versatility, achieving architecture independence through a tokenization mechanism, enabling support for… It supports CNN student models including ResNet and ConvNeXt, as well as various ViT student models. Furthermore, without changing the backbone network of the student models, it enables the student models to capture richer semantic information and fine-grained features than a single teacher model through the guidance of neural agents, and exhibits strong robustness to changes in input image resolution. In the specific downstream task of image segmentation, it achieves a comprehensive technical effect of high precision, high speed and high robustness, which emphasizes both details and global aspects, lightweight inference, and multi-source knowledge fusion. It is especially suitable for applications in fields with extremely high requirements for security and real-time performance, such as medical diagnosis, autonomous driving, and industrial quality inspection.

[0066] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0067] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A cross-domain, cross-architecture visual foundation model knowledge fusion method, characterized in that, include: Construct a heterogeneous set of teacher models and student models; The input image is fed into the student model and multiple teacher models of the heterogeneous teacher model set to obtain student visual features and teacher visual features, and all student visual features and teacher visual features are uniformly represented as a standard visual token. pass Each projection layer projects the student's standard visual token onto... The teacher standard visual token feature space corresponding to each teacher model is obtained. Student characteristic representation in a group teacher space; Introduction Each learnable proxy token, and associated with... The student feature representations in the group teacher space are combined to form Feature sequences in the group teacher space; An alternating dynamic interaction strategy is adopted for the above. The feature sequences under the group teacher space are interactively fused across space and within space; a multi-teacher global and local feature alignment strategy is adopted to align the final features after interactive fusion with the visual features of the teachers to obtain the visual global features and local features for each teacher space. The parameters of the student model and the learnable parameters of the projection layer are updated using the loss of visual global and local features for each teacher space. Iterate through the above process until the termination condition is met, thus completing the cross-domain, cross-architecture visual foundation model knowledge fusion of the student model.

2. The cross-domain, cross-architecture visual foundation model knowledge fusion method according to claim 1, characterized in that, The process of uniformly representing all student visual features and teacher visual features as a standard visual token includes: For the visual features of students and teachers, the visual features generated by different architectures are uniformly organized into a standard visual unit representation to obtain a standard visual token sequence; wherein, for CNN features, their spatial location is expanded into a set of spatial visual units; for ViT features, their patch representation or category representation is used as a sequence visual unit.

3. The cross-domain, cross-architecture visual foundation model knowledge fusion method according to claim 1, characterized in that, The passage Each projection layer projects the student's standard visual token onto... The teacher standard visual token feature space corresponding to each teacher model is obtained. Student characteristic representation in the group teacher space includes: Apply the standard visual token to each student A lightweight mapping function ,get Student characteristics in the group teacher space are represented as follows: ； in, The student model is represented in the first... The projection results in the standard visual token feature space corresponding to each teacher model; Represents a standard visual token for students; Indicates the first A lightweight mapping function.

4. The cross-domain, cross-architecture visual foundation model knowledge fusion method according to claim 1, characterized in that, The introduction Each learnable proxy token, and associated with... The student feature representations in the group teacher space are combined to form The feature sequences in the group teacher space include: Will Each learnable proxy token is concatenated with the student feature representation under the corresponding teacher space on the channel to obtain... Feature sequences in the group teacher space.

5. The cross-domain, cross-architecture visual foundation model knowledge fusion method according to claim 1, characterized in that, The aforementioned alternating dynamic interaction strategy is used to... The feature sequences under the group teacher space are interactively fused across and within the space, including: The alternating dynamic interaction strategy includes alternating cross-space interaction mechanisms and intra-space interaction mechanisms; wherein, the cross-space interaction mechanism is used to interactively integrate shared knowledge between different teacher spaces, and the intra-space interaction mechanism is used to interactively integrate specific knowledge within each teacher space. pass The alternation of cross-space interaction mechanisms and intra-space interaction mechanisms at each stage outputs the final characteristics after interaction fusion.

6. The cross-domain, cross-architecture visual foundation model knowledge fusion method according to claim 1, characterized in that, The multi-teacher global and local feature alignment strategy aligns the final features after interactive fusion with the teacher's visual features to obtain visual global and local features for each teacher's space, including: The features output by the last spatial interaction mechanism are segmented by channel to obtain visual token features and learnable token features. The learnable token features are then used to obtain a global token representation through global average pooling. Align the global token representation with the global features of the teacher model and calculate the global feature alignment loss; constrain the learnable token features through the global feature alignment loss to dynamically integrate global visual knowledge among different teachers; The fused features obtained by the alternating dynamic interaction strategy are used to optimize the visual token features of the student model in different teacher spaces, achieve local feature alignment with different teacher models, and calculate the local feature alignment loss. The final output includes visual global and local features for each teacher's space.

7. The cross-domain, cross-architecture visual foundation model knowledge fusion method according to any one of claims 1-6, characterized in that, Also includes: Three quantitative indicators are introduced: gradient cosine average similarity, gradient interference ratio, and gradient sign average difference, to quantitatively evaluate the degree of conflict between the alignment of different teacher model features and the student model parameters during the model training process.

8. A cross-domain, cross-architecture visual foundation model knowledge fusion system, characterized in that, include: The multi-model distillation framework module is used to construct a heterogeneous set of teacher models and student models. The feature unification module is used to feed the input image into the student model and multiple teacher models of the heterogeneous teacher model set to obtain student visual features and teacher visual features, and to unify the student visual features and teacher visual features into a standard visual token. The teacher represents the space construction module, which is used to... Each projection layer projects the student's standard visual token onto... The teacher standard visual token feature space corresponding to each teacher model is obtained. Student characteristic representation in the group teacher space; introduction Each learnable proxy token, and associated with... The student feature representations in the group teacher space are combined to form Feature sequences in the group teacher space; The feature fusion interaction module employs an alternating dynamic interaction strategy for the features fused together. The feature sequences under the group teacher space are interactively fused across space and within space; a multi-teacher global and local feature alignment strategy is adopted to align the final features after interactive fusion with the visual features of the teachers to obtain the visual global features and local features for each teacher space. The model optimization module utilizes the loss of visual global and local features for each teacher space to update the parameters of the student model and the learnable parameters of the projection layer, thereby completing the cross-domain and cross-architecture visual base model knowledge fusion of the student model.

9. An image segmentation method, characterized in that, include: Obtain an image segmentation visual base model that integrates cross-domain and cross-architecture knowledge, trained using the method described in any one of claims 1-7; Acquire the target image data to be processed; The target image data is input into the image segmentation visual base model, and the image segmentation result is output through model inference to achieve the segmentation of the target image.

10. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of any one of claims 1-7 or 9.

Images