A Cross-Modal Knowledge Distillation Method and System Based on Dynamic Structure Awareness

Abstract

This application provides a cross-modal knowledge distillation method and system based on dynamic structure awareness. It receives a target task dataset, divides it into a training set, and inputs the training set into a preset teacher model and a preset student model to obtain textual and visual features. Based on the parameter scale differences between the models, a structural difference index is determined in real time, and a projection matrix is ​​obtained based on the structural difference index. The corresponding transfer matrix is ​​obtained based on the real-time projection matrix, and the attention distribution information of the preset teacher model is extracted. Based on the attention distribution, knowledge distillation from the preset teacher model to the preset student model is completed, and semantic segmentation is performed on the target task dataset to obtain the semantic segmentation result. This method effectively solves the problem of difficulty in aligning network models of different depths and significantly reduces the size of the models used to process the target task.

Description

Technical Field

[0001] This application relates to the field of neural network technology, specifically to a cross-modal knowledge distillation method and system based on dynamic structure perception. Background Technology

[0002] In existing technologies, when knowledge transfer between heterogeneous models is achieved through static projection and fixed architecture matching, feature dimensions can be compressed using a projection matrix with fixed sparsity to align feature dimensions between heterogeneous models. Alternatively, the modal gap between the local receptive field of vision and the global attention mechanism of language models can be solved through the cross-modal alignment method of contrastive learning. Furthermore, the structured knowledge transfer efficiency of deep networks can be improved by matching feature distributions through optimal transfer theory.

[0003] However, compressing feature dimensions using a projection matrix with fixed sparsity makes it difficult to dynamically adapt to differences in model structure. In other words, static sparse projection experiences a sharp drop in performance under extreme parameter differences. Traditional contrastive learning methods struggle to balance intermodal structural sensitivity, and independent optimal transfer processing introduces additional computational overhead. Summary of the Invention

[0004] To address the problems in existing technologies, this application provides a cross-modal knowledge distillation method and system based on dynamic structure awareness, which can effectively solve the shortcomings of traditional technologies such as low inter-layer connection matching rate and difficulty in alignment between network models of different depths, significantly improve the alignment effect between heterogeneous models, and reduce the size of the model used to process the target task.

[0005] To solve at least one of the above problems, this application provides the following technical solution:

[0006] In a first aspect, this application provides a cross-modal knowledge distillation method based on dynamic structure awareness, comprising:

[0007] Receive the target task dataset, divide it into training sets, input the training sets into the preset teacher model to obtain text features, and input the training sets into the preset student model to obtain visual features;

[0008] Based on the parameter scale difference between the preset teacher model and the preset student model, the structural difference index is determined in real time. Based on the structural difference index, dynamic sparse projection is performed to obtain the projection matrix. The projection matrix is ​​used to align the feature dimensions of text features and the feature dimensions of visual features.

[0009] Based on the real-time projection matrix, the corresponding transfer matrix is ​​obtained, and the attention distribution information of the preset teacher model is extracted. Based on the current transfer matrix and attention distribution information, the network layers retained by the preset teacher model are determined to complete the knowledge distillation from the preset teacher model to the preset student model. Based on the preset teacher model and the preset student model that have undergone knowledge distillation, the target task dataset is semantically segmented to obtain the semantic segmentation result.

[0010] Furthermore, it also includes: determining the teacher parameter vector of the current preset teacher model, and determining the student parameter vector of the current preset student model;

[0011] Structural difference indicators are determined based on teacher parameter vectors and student parameter vectors using Euclidean norms.

[0012] The Euclidean norm is expressed as: ,in, The structural difference index is represented by θ_tea, which represents the teacher parameter vector, and θ_stu, which represents the student parameter vector. This represents the L2 norm operator.

[0013] Furthermore, it also includes: determining the basic sparsity based on the structural difference index and through sparsity calculation rules;

[0014] Determine the text feature dimension corresponding to the text features and the visual feature dimension corresponding to the visual features, and construct an initial projection matrix based on the text feature dimension, visual feature dimension and basic sparsity;

[0015] Based on the basic sparsity and text feature dimension, the projection retention value is determined. For each row of the matrix element in the initial projection matrix, the projection weight of each matrix element is determined. The matrix element corresponding to the projection weight of the largest absolute value of the projection weight is retained to obtain the retained matrix element. The values ​​of the non-retained matrix elements are reset to 0 to obtain the projection matrix.

[0016] The rules for sparsity calculation include:

[0017] Where ρ represents the fundamental sparsity, and tanh represents the hyperbolic tangent function. Indicators representing structural differences.

[0018] Furthermore, it also includes: for each visual feature, determining the degree of matching between each text feature and the current visual feature, and identifying text features with a matching degree exceeding a preset matching degree as matching text features;

[0019] Positive training sample pairs are constructed based on the current visual features and the matching text features, and negative training sample pairs are constructed based on the current visual features and the non-matching text features.

[0020] The training positive sample pairs and the training negative sample pairs are determined as training sample pairs. The cross-modal feature similarity of each training sample is determined by cross-modal feature similarity rules, which include:

[0021] Where a represents textual features and b represents visual features. Represents the vector dot product. L2 function operators for vectors;

[0022] A corresponding cost matrix is ​​constructed based on the cross-modal feature similarity. An initial transmission matrix is ​​obtained based on the cost matrix, and the initial transmission matrix is ​​processed according to a preset constraint divergence to obtain the transmission matrix.

[0023] Furthermore, after determining the cross-modal feature similarity for each training sample pair, the process also includes:

[0024] Based on the cross-modal feature similarity of each training sample pair, the corresponding structural sensitivity factor of each training sample pair is determined;

[0025] Based on the cross-modal feature similarity and structural sensitivity factor of the current training sample pair, determine the loss function of the current training sample pair;

[0026] Determine the contrastive loss function for all samples, obtain the contrastive loss function value, and adjust the model parameters of the preset student model based on the contrastive loss function to optimize the alignment quality between text features and visual features.

[0027] Furthermore, it also includes: inverting the cross-modal feature similarity and normalizing the inverted cross-modal feature similarity to obtain the corresponding cost matrix;

[0028] The cost matrix is ​​initialized by entropy regularization coefficients to obtain the initial transmission matrix. The initial transmission matrix is ​​then alternately normalized by row and column to obtain the processed transmission matrix.

[0029] Based on the cost matrix and the processing transmission matrix, the transmission distance between text features and visual features is determined by the transmission distance calculation rules, and the initial transmission matrix is ​​optimized according to the preset constraint divergence constraint and transmission distance to obtain the transmission matrix;

[0030] The rules for calculating transmission distance include:

[0031] Where D represents the transmission distance, C represents the cost matrix, Γ represents the transmission matrix, ε represents the entropy regularization coefficient, and H() represents the entropy function. This represents the inner product operator.

[0032] Furthermore, it also includes: extracting attention distribution information from the preset teacher model, and determining the importance index of each network layer in the preset teacher model based on the transfer matrix and attention distribution information;

[0033] The importance indicators are sorted from high to low to obtain the sorting results. The retention layer of the preset teacher model is determined according to the preset retention ratio and the sorting results.

[0034] Secondly, this application provides a cross-modal knowledge distillation system based on dynamic structure awareness, comprising:

[0035] The partitioning module is used to receive the target task dataset, partition it into a training set, input the training set into a preset teacher model to obtain text features, and input the training set into a preset student model to obtain visual features.

[0036] The projection module is used to determine the structural difference index in real time based on the parameter scale difference between the preset teacher model and the preset student model. Based on the structural difference index, dynamic sparse projection is performed to obtain the projection matrix. The projection matrix is ​​used to align the feature dimensions of text features and the feature dimensions of visual features.

[0037] The segmentation module is used to obtain the corresponding transfer matrix based on the real-time projection matrix, extract the attention distribution information of the preset teacher model, determine the network layers to be retained by the preset teacher model based on the current transfer matrix and attention distribution information, so as to complete the knowledge distillation from the preset teacher model to the preset student model, and perform semantic segmentation on the target task dataset based on the preset teacher model and the preset student model that have been knowledge distilled, so as to obtain the semantic segmentation result.

[0038] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the described cross-modal knowledge distillation method based on dynamic structure awareness.

[0039] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the described cross-modal knowledge distillation method based on dynamic structure awareness.

[0040] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the described cross-modal knowledge distillation method based on dynamic structure awareness.

[0041] As can be seen from the above technical solution, this application provides a cross-modal knowledge distillation method and system based on dynamic structure awareness. It innovatively receives the target task dataset and divides it into a training set. The training set is input into a preset teacher model and a preset student model to obtain textual and visual features. Based on the parameter scale differences between the preset teacher model and the preset student model, a structural difference index is determined in real time. Dynamic sparse projection is performed to obtain a projection matrix to achieve feature dimension alignment. The corresponding transfer matrix is ​​obtained based on the real-time projection matrix, and the network layers to be retained are determined by combining the attention distribution information of the preset teacher model, thus completing the knowledge distillation from the preset teacher model to the preset student model. The target task dataset is then semantically segmented based on the knowledge-distilled preset teacher model and preset student model to obtain semantic segmentation results. Thus, the feature dimensions are adaptively adjusted through dynamic sparse projection, improving the knowledge transfer efficiency under conditions of large parameter scale differences. The layer retention method based on attention distribution information improves the inter-layer connection matching rate between the preset teacher model and the preset student model. Simultaneously, the learning loss function enhances the focus on difficult samples, optimizing the knowledge distillation effect between heterogeneous models. This method effectively addresses the shortcomings of traditional techniques, such as low inter-layer connection matching rate and difficulty in alignment between network models of different depths. It significantly improves the alignment effect between heterogeneous models and reduces the size of the models used to process the target task. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the cross-modal knowledge distillation method based on dynamic structure awareness in an embodiment of this application.

[0044] Figure 2 This is a structural diagram of the cross-modal knowledge distillation system based on dynamic structure awareness in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0046] Figure label:

[0047] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0050] To effectively address the shortcomings of traditional techniques, such as low inter-layer connection matching rates and difficulty in alignment between network models of different depths, significantly improve the alignment effect between heterogeneous models, and reduce the size of the model used to process the target task, this application provides an embodiment of a cross-modal knowledge distillation method based on dynamic structure awareness. See [link to embodiment]. Figure 1 The cross-modal knowledge distillation method based on dynamic structure awareness specifically includes the following:

[0051] Step S101: Receive the target task dataset, divide the target task dataset into training sets, input the training sets into the preset teacher model to obtain text features, and input the training sets into the preset student model to obtain visual features.

[0052] Optionally, this embodiment receives a target task dataset, which may include image data and corresponding text data for semantic segmentation. A training set is obtained from the target task dataset. Data may be randomly selected from the target task dataset as the training set according to a preset training ratio, such as 70%, so that the training set can fully represent the overall distribution of the target task dataset. Alternatively, the target task dataset may be stratified and sampled to obtain the training set.

[0053] In addition, the training set is input into a preset teacher model and a preset student model. The preset teacher model processes the text data in the training set to obtain the text features corresponding to the training set, and the preset student model processes the visual data in the training set to obtain the visual features corresponding to the training set.

[0054] The preset teacher model can be an attention-based encoder / decoder (Transformer) model, and the preset student model can be a model based on a convolutional neural network architecture.

[0055] This embodiment enables the processing of training sets using preset teacher and student models to obtain textual and visual features, which can serve as the basis for cross-modal knowledge distillation and improve the alignment accuracy between heterogeneous data.

[0056] Step S102: Determine the structural difference index in real time based on the parameter scale difference between the preset teacher model and the preset student model, and obtain the projection matrix by performing dynamic sparse projection based on the structural difference index.

[0057] The projection matrix is ​​used to align the feature dimensions of text features with those of visual features.

[0058] Optionally, this embodiment determines the parameter scale difference between the preset teacher model and the preset student model. The parameter scale difference can be determined by norm calculation. The parameter scale difference is determined as a structural difference index. The sparsity of the projection matrix is ​​adaptively adjusted according to the structural difference index to generate a dynamic sparse projection matrix. The construction of the projection matrix is ​​carried out by preserving key connections to ensure the transformation of feature dimensions.

[0059] Among them, the parameter scale difference can be measured by various methods such as the number of parameters in the preset teacher model and the preset student model, and the number of neurons in each layer. The real-time structural difference index is used to indicate the degree of difference in architecture between the current preset teacher model and the current preset student model. Dynamic sparse projection can adjust the projection strategy in real time according to the parameter scale difference. That is, when the teacher model parameters of the preset teacher model and / or the student model parameters of the preset student model change, the parameter scale difference and structural difference index can be re-determined, and then the knowledge distillation from the preset teacher model to the preset student model can be completed again.

[0060] Furthermore, the feature dimensions of text features and visual features are processed by projection matrix to align the feature dimensions of text features and visual features, resulting in aligned text features and aligned visual features.

[0061] This embodiment enhances the alignment accuracy and adaptability of text and visual features through projection matrices, improves the efficiency of knowledge transfer in cross-modal knowledge distillation, enhances the generalization ability and robustness in heterogeneous architecture scenarios, and can dynamically perceive the teacher model parameters of the preset teacher model and the student parameters of the preset student model, and make corresponding adjustments based on the teacher model parameters and student model parameters, thereby improving the accuracy of cross-modal knowledge distillation.

[0062] Step S103: Obtain the corresponding transfer matrix based on the real-time projection matrix, and extract the attention distribution information of the preset teacher model. Based on the current transfer matrix and attention distribution information, determine the network layers retained by the preset teacher model to complete the knowledge distillation from the preset teacher model to the preset student model. Based on the preset teacher model and preset student model that have undergone knowledge distillation, perform semantic segmentation on the target task dataset to obtain the semantic segmentation result.

[0063] Optionally, in this embodiment, the corresponding transmission matrix is ​​obtained based on the real-time projection matrix. Training sample pairs can be established based on aligned text features and aligned visual features, the cross-modal feature similarity corresponding to each training sample pair can be determined, and the corresponding cost matrix can be constructed based on the cross-modal feature similarity. The initial transmission matrix is ​​obtained based on the cost matrix, and the initial transmission matrix is ​​processed according to the preset constraint divergence to obtain the transmission matrix.

[0064] At the same time, the attention distribution information of the preset teacher model is extracted, which reflects the degree of attention of each part of the preset teacher model in the process of processing the training set.

[0065] Based on the current transfer matrix and attention distribution information, determine the network layers to be retained in the preset teacher model. That is, select the network layer parts of the preset teacher model that are most valuable for the preset student model to learn for knowledge transfer, so as to complete the knowledge distillation from the preset teacher model to the preset student model, and transfer the effective knowledge and features in the preset teacher model to the preset student model, so that the preset student model can learn the feature expression ability of the preset teacher model.

[0066] In addition, based on the pre-defined teacher model and pre-defined student model obtained from the current knowledge distillation, semantic segmentation is performed on the target task dataset to obtain semantic segmentation results. Semantic segmentation is used to classify each pixel and other basic units in the target task dataset into the corresponding semantic category.

[0067] Furthermore, after obtaining the semantic segmentation results, conditional random fields can be applied to refine the segmentation results. By optimizing the segmentation boundaries through the spatial relationships and semantic associations between pixels, the mis-segmented regions can be reduced, making the semantic segmentation results more accurate and coherent.

[0068] This embodiment realizes the knowledge distillation process from the preset teacher model to the preset student model, which improves the model performance, feature alignment accuracy and stability in semantic segmentation tasks, enhances the adaptability to differences in feature structures of heterogeneous modalities during knowledge transfer, and improves the overall effect of cross-modal knowledge distillation and model convergence efficiency.

[0069] This embodiment utilizes the semantic knowledge of the preset teacher model while maintaining the lightweight advantage of the preset student model through dynamic sparse projection, feature alignment, and transfer matrix. This solves the feature processing problem caused by differences in model structure, improves the effect of knowledge distillation and the accuracy of semantic segmentation, and enhances the versatility and real-time performance of the preset teacher and student models that have completed knowledge distillation.

[0070] In some embodiments, structural difference indicators are determined in real time based on the parameter scale differences between a preset teacher model and a preset student model, including:

[0071] Determine the teacher parameter vector of the current preset teacher model, and determine the student parameter vector of the current preset student model;

[0072] Structural difference indicators are determined based on teacher parameter vectors and student parameter vectors using Euclidean norms.

[0073] The Euclidean norm is expressed as: ,in, The structural difference index is represented by θ_tea, which represents the teacher parameter vector, and θ_stu, which represents the student parameter vector. This represents the L2 norm operator.

[0074] Optionally, this embodiment determines the teacher parameter vector corresponding to the current preset teacher model. The teacher parameter vector is determined based on all learnable parameters of the preset teacher model. The learnable parameters include, but are not limited to, the weight matrix and bias vector of the preset teacher model. The teacher parameter vector is used to represent the knowledge and feature representation capabilities learned by the preset teacher model during training.

[0075] At the same time, the student parameter vector corresponding to the current preset student model is determined. The preset student model has a simpler structure and smaller parameter size than the preset teacher model. All learnable parameters in the preset student model are determined, and the student parameter vector is determined based on all learnable parameters in the preset student model. Although the parameter size of the preset student model is much smaller than that of the preset teacher model, the key knowledge in the preset teacher model can be inherited through effective knowledge distillation.

[0076] In addition, the structural difference index is determined based on the teacher parameter vector and the student parameter vector using the Euclidean norm, also known as the L2 norm. The Euclidean norm is a way to measure the length or size of a vector.

[0077] The Euclidean norm is expressed as: ,in, The structural difference index is represented by θ_tea, which represents the teacher parameter vector, and θ_stu, which represents the student parameter vector. θ_tea represents the Euclidean norm of the teacher parameter vector θ_tea, which is the square root of the sum of squares of the elements in θ_tea; ||θ_stu||2 represents the Euclidean norm (also known as the L2 norm) of the student parameter vector θ_stu.

[0078] The structural difference index is used to reflect the degree of structural difference between the current preset teacher model and the preset student model from the perspective of parameter size. The larger the value of the structural difference index, the larger the parameter size of the preset teacher model and the more complex the structure is relative to the preset student model. Similarly, the smaller the value of the structural difference index, the smaller the parameter size of the preset teacher model and the simpler the structure is relative to the preset student model.

[0079] Furthermore, in determining the teacher and student parameter vectors, the parameter vectors can be preprocessed using parameter pruning and quantization. Parameter pruning removes unimportant parameters (such as weights with small absolute values) from the pre-defined teacher and student models, thus simplifying the parameter vectors while retaining key parameter information. Parameter quantization converts high-precision parameter values ​​(such as 32-bit floating-point numbers) into low-precision representations (such as 8-bit integers), reducing the storage space of the parameter vectors and accelerating subsequent calculations to some extent.

[0080] For example, in processing large deep neural networks, pruning can remove about 30% to 50% of unimportant parameters, and quantization can reduce the parameter storage space to about 1 / 4 of the original size, while having a small impact on model performance.

[0081] Furthermore, in determining the structural difference indicators, other mathematical tools and indicators can be combined to conduct multi-dimensional analysis of model structural differences. For example, the cosine similarity between parameter vectors can be calculated to measure the similarity in direction between the parameter vectors of the preset teacher model and the preset student model, or the sparsity characteristics of parameter vectors can be analyzed to compare the sparsity differences in the parameter distribution of the two models.

[0082] The cosine similarity is calculated as follows:

[0083] ,in, This is used to represent the dot product of the teacher parameter vector and the student parameter vector, thus providing a more comprehensive understanding of the structural differences between the preset teacher model and the preset student model.

[0084] This embodiment improves the accuracy of feature alignment by accurately determining structural difference indicators to quantify feature alignment, training sample pair establishment, cost matrix construction, and knowledge distillation. This enables features from different modalities to be more accurately calculated and matched in a unified feature space, enhances the targeting of knowledge distillation, and allows the preset student model to learn the key knowledge and feature representation capabilities of the preset teacher model more efficiently, thereby improving the accuracy and reliability of semantic segmentation results.

[0085] In some embodiments, dynamic sparse projection based on structural difference indices is used to obtain a projection matrix, including:

[0086] Based on the structural difference index, the basic sparsity is determined by the sparsity calculation rules.

[0087] Determine the text feature dimension corresponding to the text features and the visual feature dimension corresponding to the visual features, and construct an initial projection matrix based on the text feature dimension, visual feature dimension and basic sparsity;

[0088] Based on the basic sparsity and text feature dimension, the projection retention value is determined. For each row of the matrix element in the initial projection matrix, the projection weight of each matrix element is determined. The matrix element corresponding to the projection weight of the largest absolute value of the projection weight is retained to obtain the retained matrix element. The values ​​of the non-retained matrix elements are reset to 0 to obtain the projection matrix.

[0089] The rules for sparsity calculation include:

[0090] Where ρ represents the fundamental sparsity, and tanh represents the hyperbolic tangent function. Indicators representing structural differences.

[0091] Optionally, in this embodiment, the basic sparsity ρ is determined based on the structural difference index and a sparsity calculation rule, wherein the sparsity calculation rule can be expressed as:

[0092] ;

[0093] Where ρ represents the basic sparsity, which can range from [0, 0.5], and tanh represents the hyperbolic tangent function, used to map the structural difference index to a suitable interval. The structural difference index is used to indicate the degree of difference in parameter scale between the preset teacher model and the preset student model.

[0094] The basic sparsity determines the proportion of non-zero elements retained in the projection matrix. That is, the smaller the ρ value, the sparser the projection matrix and the less feature information is retained. Similarly, the larger the ρ value, the denser the projection matrix and the more feature information is retained.

[0095] In addition, the text feature dimension corresponding to the text feature and the visual feature dimension corresponding to the visual feature are determined. The text feature dimension is used to represent the length of the text feature vector, reflecting the representational power of the text feature, while the visual feature dimension is used to represent the length of the visual feature vector, reflecting the representational power of the visual feature.

[0096] An initial projection matrix is ​​constructed based on the text feature dimension, visual feature dimension, and basic sparsity, wherein the size of the initial projection matrix is ​​the product of the text feature dimension and the visual feature dimension.

[0097] Furthermore, projection retention values ​​are determined based on the basic sparsity and text feature dimensions. For each element in the initial projection matrix, the projection weight of each element is determined. The projection weight can be understood as the importance of the element in the feature projection process. The absolute value of each element is determined, and then the k elements with the largest absolute values ​​in each row of the initial projection matrix are selected as retention elements. The values ​​of the remaining elements are reset to 0. After row-by-row processing, a sparsified projection matrix is ​​obtained. The projection matrix retains the key feature mapping relationships and reduces complexity.

[0098] Here, the projection retention value represents the number of non-zero elements retained in each row of the initial projection matrix. The expression for the projection retention value can be:

[0099] ;

[0100] Where k represents the projection retention value, d_text represents the text feature dimension, ρ represents the basic sparsity, and the projection retention value k takes the integer part, which means that the matrix elements corresponding to the k largest projection weights are retained in each row of the initial projection matrix.

[0101] Furthermore, the projection matrix has the following characteristics: each row has k non-zero matrix elements, and the remaining matrix elements are 0, which reflects sparsity;

[0102] The positions and values ​​of non-zero matrix elements are dynamically determined based on the projection weights in the initial projection matrix, which can adaptively capture the main mapping relationship between text features and visual features;

[0103] The size of the matrix is ​​the product of the text feature dimension and the visual feature dimension, which can map text features from the text feature dimension to the visual feature dimension where the visual features are located, thus achieving feature alignment.

[0104] Furthermore, the mapping fidelity of the projection matrix from the teacher model features corresponding to the preset teacher model to the preset student model features corresponding to the preset student model is verified, and the mapping error is determined to be less than the preset error threshold.

[0105] The expression for the preset error threshold is:

[0106] Where P represents the projection matrix. Used to represent the data matrix input into the preset teacher model This is used to represent the target feature matrix that the preset student model expects to achieve or align with, i.e., the output matrix of the preset teacher model. Used to represent the Frobenius norm.

[0107] Furthermore, in a multimodal environment, features from other modalities may be involved besides textual and visual features, such as speech features and depth sensor data. The dimension of the initial projection matrix can be expanded to obtain a multimodal feature fusion projection matrix, enabling it to handle the mapping relationships of various modal features. Simultaneously, the sparsity processing strategy can be optimized by combining the correlation and complementarity between different modalities, prioritizing the retention of cross-modal correlated features and improving the effectiveness of multimodal knowledge distillation.

[0108] This embodiment provides an efficient and accurate feature mapping method for cross-modal knowledge distillation through dynamic sparse projection technology. By adaptively adjusting sparsity and optimizing the feature alignment process, it improves the efficiency and effectiveness of knowledge distillation, reduces computation and storage costs, and enhances the generalization ability of the preset student model.

[0109] In some embodiments, obtaining the corresponding transmission matrix based on the real-time projection matrix includes:

[0110] For each visual feature, determine the degree of matching between each text feature and the current visual feature, and identify the text features whose matching degree exceeds the preset matching degree as matching text features;

[0111] Positive training sample pairs are constructed based on the current visual features and the matching text features, and negative training sample pairs are constructed based on the current visual features and the non-matching text features.

[0112] Training positive sample pairs and training negative sample pairs are defined as training sample pairs. The cross-modal feature similarity of each training sample is determined by cross-modal feature similarity rules, which include:

[0113] Where a represents textual features and b represents visual features. Represents the vector dot product. L2 function operators for vectors;

[0114] The corresponding cost matrix is ​​constructed based on cross-modal feature similarity. The initial transmission matrix is ​​obtained based on the cost matrix and then processed according to the preset constraint divergence to obtain the transmission matrix.

[0115] Optionally, in this embodiment, for each visual feature, the matching degree between each text feature and the current visual feature is determined, and the text features with a matching degree exceeding a preset matching degree are determined as matching text features. The matching degree can be determined by the similarity between feature vectors, or by domain knowledge or annotation information. For example, in an image-text dataset, an image may have a text describing its content. This text is semantically matched with the image, thereby determining the visual features corresponding to the image and the text features corresponding to the text as positive training sample pairs.

[0116] Similarly, for each visual feature, text features that do not match it are determined from the text features. That is, the text features are semantically unrelated to or have a very weak correlation with the visual features. These can be the remaining text features after constructing the positive samples, or text features with a matching degree not exceeding the preset matching degree, forming a negative training sample pair. The preset matching degree can be adjusted according to the specific task requirements and dataset to control the quality of the positive and negative training sample pairs.

[0117] In addition, training positive sample pairs and training negative sample pairs are defined as training sample pairs. The cross-modal feature similarity of each training sample pair is determined by the cross-modal feature similarity rule. The specific formula is as follows:

[0118] Where a represents textual features and b represents visual features. Represents the vector dot product. and Let a and b represent the L2 norms of vectors a and b, respectively.

[0119] Among them, the positive training sample pairs in the training sample pairs can be represented as Negative training sample pairs can be represented as ,in, Used to represent visual features Used to represent text features that match the current visual features. Used to represent text features that do not match the current visual features. Used to represent positive training sample pairs, and to represent semantic alignment pairs of images and text. Used to represent negative training sample pairs, representing semantic conflict pairs between images and text;

[0120] Among them, cross-modal feature similarity is used to reflect the similarity between text features and visual features in the aligned dimensional space.

[0121] Furthermore, cross-modal feature similarity can be determined through similarity metrics, or through similarity metrics models based on deep learning or geometric transformations.

[0122] In addition, a cost matrix is ​​constructed based on cross-modal feature similarity. The matrix elements in the cost matrix are used to represent the differences between the current preset teacher model and the current preset student model, which can be understood as representing the similarity or distance between the feature vectors of the preset teacher model and the feature vectors of the preset student model.

[0123] Furthermore, an initial transfer matrix is ​​obtained based on the cost matrix, and then processed according to a preset constraint divergence to obtain the final transfer matrix. The preset constraint divergence can be achieved by constraining the initial transfer matrix using preset divergence constraint rules to ensure the consistency of the transfer matrix's distribution. These preset divergence constraint rules include:

[0124] Where KL represents relative entropy, P represents the marginal distribution of the eigenvectors of the preset teacher matrix, Q represents the marginal distribution of the eigenvectors of the preset student matrix, and Th2 represents the constraint value.

[0125] The constraint value can be 0.3.

[0126] Furthermore, the threshold parameter corresponding to the preset matching degree can be dynamically adjusted according to the progress of model training and the distribution of samples. For example, in the early stage of training, a higher preset matching degree can be set to strictly screen positive training sample pairs, which can quickly learn obvious semantic matching features. As training progresses, the preset matching degree can be gradually reduced to increase the number and diversity of positive training sample pairs, enabling the model to learn a wider range of semantic relevance.

[0127] This embodiment achieves accurate determination of cross-modal feature similarity by constructing positive and negative training sample pairs, effectively improving the discriminative power of cross-modal feature alignment and the model's ability to focus on difficult samples. It also determines the cost matrix through aligned text features and aligned visual features, and further obtains the transfer matrix, which can improve the accuracy and consistency of information transmission during knowledge distillation. It can also effectively learn semantic alignment features, improve semantic understanding and discriminative ability, and enhance the adaptability and generalization performance of the preset teacher model and the preset student model.

[0128] In some embodiments, after determining the cross-modal feature similarity corresponding to each training sample through cross-modal feature similarity rules, the method further includes:

[0129] Based on the cross-modal feature similarity of each training sample pair, the corresponding structural sensitivity factor of each training sample pair is determined;

[0130] Based on the cross-modal feature similarity and structural sensitivity factor of the current training sample pair, determine the loss function of the current training sample pair;

[0131] Determine the contrastive loss function for all samples, obtain the contrastive loss function value, and adjust the model parameters of the preset student model based on the contrastive loss function to optimize the alignment quality between text features and visual features.

[0132] Optionally, after determining the cross-modal feature similarity of each training sample pair, this embodiment can determine the corresponding structural sensitivity factor of each training sample pair based on the similarity. The expression for the structural sensitivity factor includes:

[0133] ;

[0134] Where α is used to represent the structure sensitivity factor, Used to represent cross-modal feature similarity, the value range can be [0,1]. The structure sensitivity factor reflects the difficulty and importance of feature alignment of training sample pairs.

[0135] In addition, based on the cross-modal feature similarity and structural sensitivity factor of the current training sample pair, the loss function corresponding to the current training sample pair is determined, wherein the expression of the loss function includes:

[0136] ;

[0137] Where L represents the sample loss function, This indicates that the loss magnitude is adaptively adjusted using a structure-sensitive factor to enhance focus on difficult samples. Used to represent the similarity between pairs of positive training samples. The similarity of negative training sample pairs is represented by α; the structural sensitivity factor is represented by s; and the L2 norm of the special vector a or b is represented by s. .

[0138] In addition, the loss function is summarized for all training samples to obtain the comparative loss function value:

[0139] ；

[0140] in, This represents the contrastive loss function value, where N represents the total number of training sample pairs. Indicates the first The loss value for each training sample pair.

[0141] The model parameters of the preset student model can be adjusted through the backpropagation algorithm, that is, the gradient of the contrast loss function value with respect to the preset student model parameters is determined, and the student model parameters are updated through optimization algorithms (such as stochastic gradient descent or Adam optimizer) to optimize the alignment quality between text features and visual features.

[0142] Furthermore, training weights can be assigned to each training sample pair, and the contrastive loss function value can be determined based on the training weights and the training sample pair loss function. The training weights are determined based on the rarity, importance, and difficulty of the training sample pair.

[0143] Furthermore, the aligned text features and visual features can be reduced to a low-dimensional space using t-SNE. The degree of overlap between the text features and visual features can be determined in the low-dimensional embedding space. Text features and visual features with an overlap greater than a preset overlap threshold can be identified as high-quality text features and visual features.

[0144] This embodiment introduces a structure-sensitive factor and an adaptively adjusted loss function, enabling the preset teacher model and preset student model to pay more attention to difficult training sample pairs, improve the learning effect on difficult training samples, accelerate the convergence speed, enhance the generalization ability, and more accurately handle complex semantic interactions in multimodal data processing, thereby improving the performance of multimodal tasks.

[0145] In some embodiments, a corresponding cost matrix is ​​constructed based on cross-modal feature similarity, an initial transfer matrix is ​​obtained based on the cost matrix, and the initial transfer matrix is ​​processed according to a preset constraint divergence to obtain the transfer matrix, including:

[0146] Invert the cross-modal feature similarity and normalize the inverted cross-modal feature similarity to obtain the corresponding cost matrix;

[0147] The cost matrix is ​​initialized by entropy regularization coefficients to obtain the initial transmission matrix. The initial transmission matrix is ​​then alternately normalized by row and column to obtain the processed transmission matrix.

[0148] Based on the cost matrix and the processing transmission matrix, the transmission distance between text features and visual features is determined by the transmission distance calculation rules, and the initial transmission matrix is ​​optimized according to the preset constraint divergence constraint and transmission distance to obtain the transmission matrix;

[0149] The rules for calculating transmission distance include:

[0150] Where D represents the transmission distance, C represents the cost matrix, Γ represents the transmission matrix, ε represents the entropy regularization coefficient, and H() represents the entropy function. This represents the inner product operator.

[0151] Optionally, this embodiment performs an inversion operation on the cross-modal feature similarity, that is, calculates... ,in This represents the cross-modal feature similarity. The inverted value reflects the degree of cross-modal dissimilarity between features. The inverted cross-modal feature similarity is normalized to ensure that all values ​​fall within the range of [0,1].

[0152] The initial transfer matrix is ​​obtained by initializing the cost matrix using entropy regularization coefficients. The initialization process can be represented as follows:

[0153] ;

[0154] in, Let C represent the initial transfer matrix, exp() represent the exponential function, and C ij Let represent the cost matrix, and ε represent the entropy regularization coefficient. The entropy regularization coefficient is used to control the entropy of the initial transfer matrix, and it is usually taken as a small value (such as 0.05) to ensure that the initial transfer matrix has a certain degree of sparsity and smoothness.

[0155] The initial transmission matrix is ​​alternately normalized by row and column to obtain the processed transmission matrix. Row normalization means normalizing the elements of each row of the matrix so that their sum is 1, and column normalization means normalizing the elements of each column so that their sum is 1.

[0156] Alternately perform row normalization and column normalization until the processing transfer matrix converges. This requires multiple iterations until the change in the processing transfer matrix is ​​less than a preset change threshold (e.g., 10). −6 ).

[0157] The alternating normalization process can be represented as: ,in, The processing transfer matrix is ​​represented by Diag(), the diagonal matrix is ​​represented by P, the edge distribution of text features is represented by Q, and the edge distribution of visual features is represented by Q.

[0158] Furthermore, based on the cost matrix and the processing transfer matrix, the transfer distance between text features and visual features can be determined through a transfer distance calculation rule, which can be expressed as:

[0159] ;

[0160] in, Let C represent the Frobenius inner product of the cost matrix C and the transmission matrix Γ, let Γ represent the total transmission cost, let ε represent the entropy regularization coefficient, and let H(Γ) represent the entropy of processing the transmission matrix.

[0161] In addition, the initial transmission matrix is ​​optimized according to the preset constraint divergence (such as KL divergence) and transmission distance to obtain the transmission matrix Γ.

[0162] The optimization process typically uses optimization algorithms such as gradient descent to minimize the transmission distance while ensuring that the transmission matrix meets the preset constraint divergence conditions.

[0163] This embodiment ensures the effective transfer of knowledge from the pre-defined teacher model through the constructed cost matrix and optimized transfer matrix, thereby improving the feature alignment accuracy and learning efficiency of the pre-defined student model. In multimodal data processing, the ability to process complex multimodal data is further enhanced by introducing the construction of a transfer matrix for multimodal feature fusion, dynamic constraint divergence adjustment, and transfer matrix optimization combined with sample weights.

[0164] In some embodiments, attention distribution information of a preset teacher model is extracted, and the network layers to be retained in the preset teacher model are determined based on the transfer matrix and attention distribution information, including:

[0165] Extract attention distribution information from the pre-defined teacher model, and determine the importance index of each network layer in the pre-defined teacher model based on the transfer matrix and attention distribution information;

[0166] The importance indicators are sorted from high to low to obtain the sorting results. The retention layer of the preset teacher model is determined according to the preset retention ratio and the sorting results.

[0167] Optionally, this embodiment extracts the attention distribution information of the preset teacher model to obtain the attention weights and distribution of each network layer when the preset teacher model processes input data. The attention distribution information reflects the degree of attention and focus area of ​​the preset teacher model to the input features at different levels. That is, for each network layer in the preset teacher model, its attention weight matrix is ​​calculated. The weight matrix represents the degree of attention of the network layer to different input features.

[0168] For example, attention distribution information of each network layer can be obtained by extracting the attention weight matrix in the self-attention mechanism, which represents the degree of attention paid to different words and features in the text by the pre-defined teacher model when processing text data.

[0169] In addition, based on the extracted attention distribution information and transfer matrix, the importance index of each network layer in the preset teacher model is determined. The transfer matrix reflects the transfer plan and importance of the feature vectors from the preset teacher model to the feature vectors of the preset student model.

[0170] The method for determining the importance index for each network layer in the pre-defined teacher model is as follows:

[0171] For the l-th network layer of the teacher model, determine its average attention weight Attn. l It can be obtained by averaging all elements of the attention weight matrix of the network layer;

[0172] By combining the transfer matrix, the importance score (Imp) of the network layer's features in the knowledge distillation process is determined. l This can be achieved by determining the sum of the weights of the network layer features in the transmission matrix.

[0173] In addition, Imp is an important metric for all network layers. l The layers are sorted from highest to lowest importance to determine the key layers for the knowledge distillation process.

[0174] The retention layers of the preset teacher model are determined according to the preset retention ratio and ranking results. For example, if the preset retention ratio is 0.5, the network layers ranked in the top 50% are retained. The retention layers will be the focus of retention and utilization during the knowledge distillation process to ensure that key knowledge can be effectively transferred to the preset student model. If the number of retention layers obtained according to the retention ratio is not an integer, the calculated number of retention layers is rounded up.

[0175] This embodiment achieves the determination of importance indicators of each network layer in the preset teacher model by accurately extracting attention distribution information and transfer matrix, and dynamically adjusting the retention ratio, which can effectively improve the efficiency of knowledge distillation and the performance of the preset student model, while reducing the complexity of the model.

[0176] To effectively address the shortcomings of traditional techniques, such as low inter-layer connection matching rates and difficulty in alignment between network models of different depths, significantly improve the alignment effect between heterogeneous models, and reduce the size of the model used to process the target task, this application provides an embodiment of a dynamic structure-aware cross-modal knowledge distillation system for implementing all or part of the aforementioned dynamic structure-aware cross-modal knowledge distillation. See [link to embodiment]. Figure 2 The cross-modal knowledge distillation system based on dynamic structure awareness specifically includes the following:

[0177] The partitioning module 10 is used to receive the target task dataset, partition the target task dataset to obtain a training set, input the training set into a preset teacher model to obtain text features, and input the training set into a preset student model to obtain visual features.

[0178] Projection module 20 is used to determine the structural difference index in real time based on the parameter scale difference between the preset teacher model and the preset student model, and to obtain the projection matrix by dynamic sparse projection based on the structural difference index. The projection matrix is ​​used to align the feature dimensions of text features and the feature dimensions of visual features.

[0179] The segmentation module 30 is used to obtain the corresponding transmission matrix based on the real-time projection matrix, extract the attention distribution information of the preset teacher model, determine the network layer retained by the preset teacher model based on the current transmission matrix and attention distribution information, so as to complete the knowledge distillation from the preset teacher model to the preset student model, and perform semantic segmentation on the target task dataset based on the preset teacher model and the preset student model that have been knowledge distilled, so as to obtain the semantic segmentation result.

[0180] As described above, the cross-modal knowledge distillation system based on dynamic structure awareness provided in this application can innovatively receive the target task dataset and divide it into training sets. The training sets are then input into a preset teacher model and a preset student model to obtain textual and visual features. Based on the parameter scale differences between the preset teacher model and the preset student model, a structural difference index is determined. Dynamic sparse projection is performed to obtain a projection matrix to achieve feature dimension alignment. The corresponding transfer matrix is ​​obtained based on the real-time projection matrix, and the network layers to be retained are determined by combining the attention distribution information of the preset teacher model. This completes the knowledge distillation from the preset teacher model to the preset student model. The target task dataset is then semantically segmented based on the knowledge-distilled preset teacher model and the preset student model to obtain semantic segmentation results. Thus, the feature dimensions are adaptively adjusted through dynamic sparse projection, improving the knowledge transfer efficiency under conditions of large parameter scale differences. The layer retention method based on attention distribution information improves the inter-layer connection matching rate between the preset teacher model and the preset student model. Simultaneously, the learning loss function enhances the focus on difficult samples, optimizing the knowledge distillation effect between heterogeneous models. This method effectively addresses the shortcomings of traditional techniques, such as low inter-layer connection matching rate and difficulty in alignment between network models of different depths. It significantly improves the alignment effect between heterogeneous models and reduces the size of the models used to process the target task.

[0181] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies, such as low inter-layer connection matching rates and difficulty in alignment between network models of different depths, significantly improve the alignment effect between heterogeneous models, and reduce the size of the model used to process the target task, this application provides an embodiment of an electronic device for implementing all or part of the aforementioned cross-modal knowledge distillation method based on dynamic structure awareness. The electronic device specifically includes the following components:

[0182] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the dynamic structure-aware cross-modal knowledge distillation system and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the dynamic structure-aware cross-modal knowledge distillation method and the embodiments of the dynamic structure-aware cross-modal knowledge distillation system in the embodiments, the content of which is incorporated herein, and repeated details will not be described again.

[0183] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0184] In practical applications, parts of the cross-modal knowledge distillation method based on dynamic structure awareness can be executed on the electronic device side as described above, or all operations can be completed in the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0185] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0186] Figure 3 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 3 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0187] In one embodiment, the functionality of the cross-modal knowledge distillation method based on dynamic structure awareness can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0188] Step S101: Receive the target task dataset, divide the target task dataset into training sets, input the training sets into the preset teacher model to obtain text features, and input the training sets into the preset student model to obtain visual features;

[0189] Step S102: Determine the structural difference index in real time based on the parameter scale difference between the preset teacher model and the preset student model, and obtain the projection matrix by dynamic sparse projection based on the structural difference index. The projection matrix is ​​used to align the feature dimensions of text features and the feature dimensions of visual features.

[0190] Step S103: Obtain the corresponding transfer matrix based on the real-time projection matrix, and extract the attention distribution information of the preset teacher model. Based on the current transfer matrix and attention distribution information, determine the network layers retained by the preset teacher model to complete the knowledge distillation from the preset teacher model to the preset student model. Based on the preset teacher model and preset student model that have undergone knowledge distillation, perform semantic segmentation on the target task dataset to obtain the semantic segmentation result.

[0191] As described above, the electronic device provided in this application innovatively receives the target task dataset and divides it into a training set. The training set is then input into a preset teacher model and a preset student model to obtain textual and visual features. Based on the parameter scale differences between the preset teacher model and the preset student model, a structural difference index is determined. Dynamic sparse projection is performed to obtain a projection matrix to achieve feature dimension alignment. The corresponding transfer matrix is ​​obtained based on the real-time projection matrix, and the network layers to be retained are determined by combining the attention distribution information of the preset teacher model. This completes knowledge distillation from the preset teacher model to the preset student model. The target task dataset is then semantically segmented based on the knowledge-distilled preset teacher model and the preset student model to obtain semantic segmentation results. Thus, by adaptively adjusting the feature dimensions through dynamic sparse projection, the knowledge transfer efficiency is improved under conditions of large parameter scale differences. The layer retention method based on attention distribution information improves the inter-layer connection matching rate between the preset teacher model and the preset student model. Simultaneously, the learning loss function enhances the focus on difficult samples, optimizing the knowledge distillation effect between heterogeneous models. This method effectively addresses the shortcomings of traditional techniques, such as low inter-layer connection matching rate and difficulty in alignment between network models of different depths. It significantly improves the alignment effect between heterogeneous models and reduces the size of the models used to process the target task.

[0192] In another embodiment, the dynamic structure-aware cross-modal knowledge distillation system can be configured separately from the central processing unit 9100. For example, the dynamic structure-aware cross-modal knowledge distillation system can be configured as a chip connected to the central processing unit 9100, and the function of the dynamic structure-aware cross-modal knowledge distillation method can be realized through the control of the central processing unit.

[0193] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3 For components not shown, please refer to existing technologies.

[0194] like Figure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0195] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0196] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0197] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0198] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0199] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0200] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0201] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the dynamic structure-aware cross-modal knowledge distillation method with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the dynamic structure-aware cross-modal knowledge distillation method with a server or client as the execution subject in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0202] Step S101: Receive the target task dataset, divide the target task dataset into training sets, input the training sets into the preset teacher model to obtain text features, and input the training sets into the preset student model to obtain visual features;

[0203] Step S102: Determine the structural difference index in real time based on the parameter scale difference between the preset teacher model and the preset student model, and obtain the projection matrix by dynamic sparse projection based on the structural difference index. The projection matrix is ​​used to align the feature dimensions of text features and the feature dimensions of visual features.

[0204] Step S103: Obtain the corresponding transfer matrix based on the real-time projection matrix, and extract the attention distribution information of the preset teacher model. Based on the current transfer matrix and attention distribution information, determine the network layers retained by the preset teacher model to complete the knowledge distillation from the preset teacher model to the preset student model. Based on the preset teacher model and preset student model that have undergone knowledge distillation, perform semantic segmentation on the target task dataset to obtain the semantic segmentation result.

[0205] As described above, the computer-readable storage medium provided in this application innovatively receives the target task dataset and divides it into a training set. The training set is then input into a preset teacher model and a preset student model to obtain textual and visual features. A structural difference index is determined based on the parameter scale differences between the preset teacher model and the preset student model. Dynamic sparse projection is performed to obtain a projection matrix to achieve feature dimension alignment. The corresponding transfer matrix is ​​obtained based on the real-time projection matrix, and the network layers to be retained are determined by combining the attention distribution information of the preset teacher model. This completes knowledge distillation from the preset teacher model to the preset student model. The target task dataset is then semantically segmented based on the knowledge-distilled preset teacher model and preset student model to obtain semantic segmentation results. Dynamic sparse projection adaptively adjusts the feature dimensions, improving knowledge transfer efficiency under conditions of large parameter scale differences. The layer retention method based on attention distribution information improves the inter-layer connection matching rate between the preset teacher model and the preset student model. Simultaneously, the learning loss function enhances the focus on difficult samples, optimizing the knowledge distillation effect between heterogeneous models. This method effectively addresses the shortcomings of traditional techniques, such as low inter-layer connection matching rate and difficulty in alignment between network models of different depths. It significantly improves the alignment effect between heterogeneous models and reduces the size of the models used to process the target task.

[0206] Embodiments of this application also provide a computer program product capable of implementing all steps of the dynamic structure-aware cross-modal knowledge distillation method with the execution subject being a server or client in the above embodiments. When executed by a processor, this computer program / instruction implements the steps of the dynamic structure-aware cross-modal knowledge distillation method. For example, the computer program / instruction implements the following steps:

[0207] Step S101: Receive the target task dataset, divide the target task dataset into training sets, input the training sets into the preset teacher model to obtain text features, and input the training sets into the preset student model to obtain visual features;

[0208] Step S102: Determine the structural difference index in real time based on the parameter scale difference between the preset teacher model and the preset student model, and obtain the projection matrix by dynamic sparse projection based on the structural difference index. The projection matrix is ​​used to align the feature dimensions of text features and the feature dimensions of visual features.

[0209] Step S103: Obtain the corresponding transfer matrix based on the real-time projection matrix, and extract the attention distribution information of the preset teacher model. Based on the current transfer matrix and attention distribution information, determine the network layers retained by the preset teacher model to complete the knowledge distillation from the preset teacher model to the preset student model. Based on the preset teacher model and preset student model that have undergone knowledge distillation, perform semantic segmentation on the target task dataset to obtain the semantic segmentation result.

[0210] As described above, the computer program product provided in this application innovatively receives the target task dataset and divides it into a training set. The training set is then input into a preset teacher model and a preset student model to obtain textual and visual features. Based on the parameter scale differences between the preset teacher model and the preset student model, a structural difference index is determined. Dynamic sparse projection is performed to obtain a projection matrix to achieve feature dimension alignment. The corresponding transfer matrix is ​​obtained based on the real-time projection matrix, and the network layers to be retained are determined by combining the attention distribution information of the preset teacher model. This completes knowledge distillation from the preset teacher model to the preset student model. Based on the knowledge-distilled preset teacher model and preset student model, semantic segmentation is performed on the target task dataset to obtain semantic segmentation results. Thus, by adaptively adjusting the feature dimensions through dynamic sparse projection, the knowledge transfer efficiency is improved under conditions of large parameter scale differences. The layer retention method based on attention distribution information improves the inter-layer connection matching rate between the preset teacher model and the preset student model. Simultaneously, the learning loss function enhances the focus on difficult samples, optimizing the knowledge distillation effect between heterogeneous models. This method effectively addresses the shortcomings of traditional techniques, such as low inter-layer connection matching rate and difficulty in alignment between network models of different depths. It significantly improves the alignment effect between heterogeneous models and reduces the size of the models used to process the target task.

[0211] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0213] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0215] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A cross-modal knowledge distillation method based on dynamic structure awareness, characterized in that, The method includes: Receive the target task dataset, divide the target task dataset into a training set, input the training set into a preset teacher model to obtain text features, and input the training set into a preset student model to obtain visual features; The process involves determining the teacher parameter vector of the current preset teacher model and the student parameter vector of the current preset student model. Using the Euclidean norm, a structural difference index is determined in real-time based on the teacher and student parameter vectors. Based on the structural difference index, a basic sparsity is determined using sparsity calculation rules. The text feature dimension corresponding to the text features and the visual feature dimension corresponding to the visual features are determined. An initial projection matrix is ​​constructed based on the text feature dimension, the visual feature dimension, and the basic sparsity. Projection retention values ​​are determined based on the basic sparsity and the text feature dimension. For each row of the initial projection matrix, the projection weight of each matrix element is determined. The matrix element corresponding to the projection weight with the largest absolute value among the projection weights is retained to obtain the retained matrix elements. The values ​​of non-retained matrix elements are reset to 0 to obtain the projection matrix. The Euclidean norm expression is: ,in, Let θ_tea represent the structural difference index, θ_stu represent the teacher parameter vector, and θ_stu represent the student parameter vector. The L2 norm operator is represented, and the sparsity calculation rules include: ,in, The basic sparsity is represented by tanh, where tanh represents the hyperbolic tangent function. The structural difference index is represented by the projection matrix, which is used to align the feature dimensions of the text features and the feature dimensions of the visual features. The corresponding transfer matrix is ​​obtained based on the real-time projection matrix, and the attention distribution information of the preset teacher model is extracted. Based on the current transfer matrix and the attention distribution information, the network layers retained by the preset teacher model are determined to complete the knowledge distillation from the preset teacher model to the preset student model. Based on the currently knowledge-distilled preset teacher model and the preset student model, semantic segmentation is performed on the target task dataset to obtain the semantic segmentation result.

2. The method according to claim 1, characterized in that, The process of obtaining the corresponding transmission matrix based on the real-time projection matrix includes: For each visual feature, determine the degree of matching between each text feature and the current visual feature, and determine the text features whose degree of matching exceeds a preset degree of matching as matching text features; A training positive sample pair is constructed based on the current visual features and the matching text features, and a training negative sample pair is constructed based on the current visual features and the non-matching text features. The training positive sample pairs and the training negative sample pairs are determined as training sample pairs. The cross-modal feature similarity corresponding to each training sample is determined by cross-modal feature similarity rules, which include: Where a represents the text feature and b represents the visual feature. Represents the vector dot product. L2 function operators for vectors; A corresponding cost matrix is ​​constructed based on the cross-modal feature similarity. An initial transmission matrix is ​​obtained based on the cost matrix, and the initial transmission matrix is ​​processed according to a preset constraint divergence to obtain the transmission matrix.

3. The method according to claim 2, characterized in that, After determining the cross-modal feature similarity corresponding to each training sample through cross-modal feature similarity rules, the method further includes: Based on the cross-modal feature similarity of each training sample pair, the corresponding structural sensitivity factor of each training sample pair is determined; Based on the cross-modal feature similarity and the structural sensitivity factor of the current training sample pair, determine the training sample pair loss function corresponding to the current training sample pair; Determine the loss function for all training sample pairs, obtain the contrastive loss function value, and adjust the model parameters of the preset student model based on the contrastive loss function to optimize the alignment quality between the text features and the visual features.

4. The method according to claim 2, characterized in that, The step of constructing a corresponding cost matrix based on the cross-modal feature similarity, obtaining an initial transfer matrix based on the cost matrix, and processing the initial transfer matrix according to a preset constraint divergence to obtain the transfer matrix includes: The cross-modal feature similarity is inverted and normalized to obtain the corresponding cost matrix; The cost matrix is ​​initialized by entropy regularization coefficient to obtain an initial transmission matrix. The initial transmission matrix is ​​then alternately normalized by row and column to obtain a processed transmission matrix. Based on the cost matrix and the processing transmission matrix, the transmission distance between the text features and the visual features is determined by the transmission distance calculation rules, and the initial transmission matrix is ​​optimized according to the preset constraint divergence constraint and the transmission distance to obtain the transmission matrix; The transmission distance calculation rules include: Where D represents the transmission distance, C represents the cost matrix, and Γ represents the transmission matrix. H represents the entropy regularity coefficient, and H() represents the entropy function. This represents the inner product operator.

5. The method according to claim 1, characterized in that, The step of extracting the attention distribution information of the preset teacher model and determining the network layers to be retained in the preset teacher model based on the current transfer matrix and the attention distribution information includes: Extract the attention distribution information of the preset teacher model, and determine the importance index of each network layer in the preset teacher model based on the transfer matrix and the attention distribution information; The importance indicators are sorted from high to low to obtain the sorting results. The retention layer of the preset teacher model is determined according to the preset retention ratio and the sorting results.

6. A cross-modal knowledge distillation system based on dynamic structure awareness, characterized in that, The system includes: The partitioning module is used to receive the target task dataset, partition the target task dataset to obtain a training set, input the training set into a preset teacher model to obtain text features, and input the training set into a preset student model to obtain visual features; The projection module is used to determine the teacher parameter vector of the current preset teacher model and the student parameter vector of the current preset student model. Based on the teacher and student parameter vectors, a structural difference index is determined in real time using the Euclidean norm. Based on the structural difference index, a basic sparsity is determined using sparsity calculation rules. The text feature dimension corresponding to the text features and the visual feature dimension corresponding to the visual features are determined. An initial projection matrix is ​​constructed based on the text feature dimension, the visual feature dimension, and the basic sparsity. Projection retention values ​​are determined based on the basic sparsity and the text feature dimension. For each row of the initial projection matrix, the projection weight of each matrix element is determined. The matrix element corresponding to the projection weight with the largest absolute value of the projection weight is retained to obtain the retained matrix elements. The values ​​of non-retained matrix elements are reset to 0 to obtain the projection matrix. The Euclidean norm expression is: ,in, Let θ_tea represent the structural difference index, θ_stu represent the teacher parameter vector, and θ_stu represent the student parameter vector. The L2 norm operator is represented, and the sparsity calculation rules include: ,in, The basic sparsity is represented by tanh, where tanh represents the hyperbolic tangent function. The structural difference index is represented by the projection matrix, which is used to align the feature dimensions of the text features and the feature dimensions of the visual features. The segmentation module is used to obtain the corresponding transfer matrix based on the real-time projection matrix, extract the attention distribution information of the preset teacher model, determine the network layers to be retained by the preset teacher model based on the current transfer matrix and the attention distribution information, so as to complete the knowledge distillation from the preset teacher model to the preset student model, and perform semantic segmentation on the target task dataset based on the currently knowledge-distilled preset teacher model and the preset student model to obtain the semantic segmentation result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the cross-modal knowledge distillation method based on dynamic structure awareness as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the cross-modal knowledge distillation method based on dynamic structure awareness as described in any one of claims 1 to 5.

Images