Synchronous transfer learning method for network architecture search on multimodal graph data

Abstract

This application discloses a network architecture search synchronous transfer learning method for multimodal graph data. In this method, the collected near-infrared spectrum, Raman spectrum and microscopic image are pre-processed and then feature extracted respectively, and the extracted features are fused through a cross-attention mechanism to form a new optimal fusion feature; in the constructed SSL_NST module, the fused data set is divided into a source domain and a target domain through transfer learning technology, and the neural network architecture search NAS adopts a three-stage optimization strategy including network search, fine-tuning and feedback; a loss weighted optimization strategy is introduced into the obtained target domain model to adjust the ratio between the classification loss value and the sample weight to optimize the training process of the fusion data model. This application solves the technical problems of low efficiency and poor method of graph fusion in related technologies, and the inability to build an effective model when there is insufficient labeled sample data.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal data fusion analysis, and more specifically, to a network architecture search and synchronous transfer learning method for multimodal graph data. Background Art

[0002] In the agricultural field, near-infrared spectroscopy, Raman spectroscopy and microscopic image detection technologies have been widely studied and applied due to their portability and efficiency.

[0003] However, the current fusion efficiency of spectra and images is low and the model methods are poor. There is significant heterogeneity between the two in data dimensions and feature space. Traditional fusion methods are difficult to break through the modal gap, resulting in insufficient utilization of complementary information. Feature expression has the dual problems of dimensionality disaster and semantic fault, which makes it difficult to meet the actual needs of high precision and low labeling cost in agricultural spectral detection.

[0004] In addition, mainstream machine learning and deep learning methods rely heavily on large-scale labeled data during model training and verification. However, insufficient labeled data has become the main bottleneck in building high-precision models.

[0005] Therefore, when graph fusion efficiency is low, methods are poor, and labeled sample data is insufficient, how to efficiently build an effective model becomes a difficult problem in the field of multimodal graph fusion modeling and analysis. Summary of the Invention

[0006] This application provides a network architecture search and synchronous transfer learning method for multimodal graph data, which at least solves the technical problems of low efficiency and poor methods of graph fusion in related technologies, and the inability to build an effective model when there is insufficient labeled sample data.

[0007] According to one aspect of the present application, a network architecture search and synchronous transfer learning method for multimodal graph data is provided, the migration method comprising the following steps:

[0008] Perform feature extraction and fusion on the collected near-infrared spectra, Raman spectra and microscopic images;

[0009] In the constructed SSL_NST module, the fused dataset is divided into source and target domains through transfer learning. The neural network architecture search (NAS) adopts a three-stage optimization strategy including network search, fine-tuning, and feedback.

[0010] In the network search stage, the default search space of the neural network architecture search NAS is replaced with the CNN and RNN combined architecture space, and the neural network architecture search NAS is used to search for the source domain model on the source domain data; in the fine-tuning stage, the target domain data is assigned pseudo-labels based on the source domain model using a semi-supervised self-training algorithm, and the maximum entropy sampling in active learning is used to filter out the pseudo-label data, freeze the network structure of the source domain model, and use the filtered pseudo-label data to fine-tune the source domain model. By unfreezing the network layer layer by layer and adjusting the optimizer and learning rate, the target domain loss is minimized; in the feedback stage, the loss of the fine-tuned model on the target domain task is used as a feedback signal to act on the search strategy of the network search and guide the next round of search process; after a preset number of iterations, the searched network structure is used as the target domain model for the target domain task.

[0011] Optionally, the step of extracting and fusing features of the collected near-infrared spectrum, Raman spectrum, and microscopic image includes:

[0012] The acquired near-infrared spectra, Raman spectra and microscopic images were normalized using Min-Max normalization and pre-processed using multivariate scattering correction (MSC);

[0013] The preprocessed near-infrared spectrum and Raman spectrum are respectively extracted with the self-attention mechanism to obtain the 128-dimensional optimal features F-nir and F-raman;

[0014] The 128-dimensional optimal features of near-infrared spectroscopy and Raman spectroscopy are concatenated to form a 256-dimensional fusion feature F-nir-raman;

[0015] The image size of the preprocessed microscopic image is resized to 224×224 pixels and the self-attention mechanism is used to extract the 256-dimensional optimal feature F-micro image;

[0016] The 256-dimensional fusion feature F-nir-raman and the 256-dimensional optimal feature F-micro image are fused using the cross attention mechanism to form a 512-dimensional optimal fusion feature F-fusion.

[0017] Optionally, the step of extracting the 256-dimensional optimal feature F-micro image using the self-attention mechanism includes:

[0018] Set the input feature sequence X∈R n×d When extracting 128-dimensional features from near-infrared spectroscopy and Raman spectroscopy, the learnable weight matrix W is first used. Q 、W K 、W V ∈R d×128, map the input features X to 128-dimensional query vectors Q=XW Q , key vector K=XW K Sum vector V=XW V ; Then calculate the dot product of the query vector and all key vectors and divide by the square root of the dimension to scale, the scaling factor is , and then generate the attention weight matrix through the Softmax function, which represents the degree of correlation between the features at each position in the input sequence; finally, multiply the attention weight by the value vector to obtain the 128-dimensional self-attention output feature that integrates the internal dependencies of the sequence ,

[0019] When extracting 256-dimensional features from microscopic images, the input features are mapped into 256-dimensional query, key, and value vectors; the dimension scaling factor is adjusted to , generating 256-dimensional self-attention output features , expressed as:

[0020] 128-dimensional self-attention output features and 256-dimensional self-attention output features Respectively expressed as:

[0021]

[0022]

[0023] Where X represents the input feature, represents the weight matrix of the value vector V, represents the weight matrix of the query vector Q, represents the weight matrix of the key vector K, and T represents the transpose of the matrix.

[0024] Optionally, the step of using the cross attention mechanism to perform feature fusion to form a 512-dimensional optimal fusion feature F-fusion includes:

[0025] set up It is the 256-dimensional fusion feature after the near-infrared spectrum and Raman spectrum are spliced ​​together. The 256-dimensional features extracted from the microscopic image are first passed through the learnable weight matrix 、 、 , spectral fusion features Mapping to query vector , the image features Mapping to key vector Sum value vector ;

[0026] Then the dot product of the spectral query vector and the image key vector is calculated and scaled, and the cross-modal attention weight is generated through Softmax, which represents the dependency of the spectral feature on the image feature;

[0027] Multiply the attention weight by the image value vector to obtain the response characteristics of the image features to the spectral features;

[0028] Finally, the original spectral fusion feature F3 and the cross-modal response feature are concatenated along the feature dimension through the Concat operation to generate a 512-dimensional cross-modal fusion feature F5. The 512-dimensional fusion feature F5 output by the cross-attention mechanism can be expressed as:

[0029]

[0030] Where F3 represents the spectral fusion feature, represents the learnable weight matrix of the query vector, F4 represents the image features, represents the learnable weight matrix of the key vector, Represents the learnable weight matrix of the value vector, and T represents the transpose of the matrix.

[0031] Optionally, the step of assigning pseudo labels to the target domain data using a semi-supervised self-training algorithm and filtering out pseudo-labeled data using maximum entropy sampling in active learning includes:

[0032] Set the fusion dataset to , where x i represents the sample of the fused dataset, y i Indicates the label corresponding to the sample, n is the number of labeled samples; the source domain dataset is , where m is the number of source domain samples, x j Represents samples of the source domain dataset;

[0033] The neural network model generated by the NAS architecture is used as the classification model S. Based on the data distribution P(x, y) of the source domain data, the model uses the confidence threshold mechanism in the self-training algorithm to screen the target domain samples. First, the model C is used to classify the target domain unlabeled sample set D. u Each sample x in j Make predictions and get prediction results and its corresponding prediction confidence(x j ); Then set a confidence threshold τ, and compare the prediction confidence of each sample with the threshold to filter out samples with confidence higher than the threshold. The filtered target domain sample set D t By all satisfying confidence(x j )>τ conditional sample x jComposition, the selected target domain sample set D t Expressed as:

[0034] ;

[0035] Where x j represents the sample, D u represents the sample set, τ represents the confidence threshold;

[0036] Then, the information entropy of the selected target domain samples is calculated. In the process of calculating the information entropy, the conditional probability p(y c |x), then take the natural logarithm of the conditional probability of each category, multiply it by the conditional probability and take the negative value to get the uncertainty contribution of a single category; finally, sum the uncertainty contributions corresponding to all categories to get the information entropy H(x) of sample x, where the target domain sample x j The information entropy is expressed as:

[0037]

[0038] Where c represents the category of the target domain sample x, c=1,2,...,C, C is the total number of categories; p(y c |x) represents the conditional probability, that is, the posterior probability that sample x belongs to category c under given features;

[0039] Finally, in each iteration t, the target domain data set D is sampled based on the idea of ​​uncertain maximum entropy sampling in active learning. t Sampling is performed based on the sampling function S(D t ) Get the required sample subset D v =S(D t ), and these samples are labeled by the source domain model and given pseudo labels .

[0040] Preferably, the step of searching for a source domain model on the source domain data using a neural network architecture search (NAS) comprises:

[0041] In the entire search process, the first stage of the search is to use Gaussian process to search the candidate network architecture in the source domain data D u The prediction loss on is modeled as:

[0042]

[0043] Where u(·) represents the mean of the Gaussian process prediction loss, σ(·) represents the standard deviation of the Gaussian process prediction loss, β represents the balance factor, and D u represents the source domain dataset, Cost(f, D u) represents the network architecture f in the source domain data D u The prediction loss on ; the degree of attention to the mean and uncertainty of model performance is adjusted by the balance factor β:

[0044] When β is large, the search process tends to explore architectures with high uncertainty;

[0045] When β is small, the search process focuses more on verified low-loss architectures;

[0046] Generate candidate architectures in the first stage Based on the minimization of candidate architectures In the target domain data D t The prediction loss on , filter out the network architecture that best suits the target domain ,This optimization goal is directly related to the target domain performance, guiding the NAS process to search in the direction of stronger cross-domain adaptability, and realizing the synchronous and stable migration of the model structure. The optimization goal is expressed as: , where Representing candidate architectures In the target domain data D t The prediction loss on represents the candidate architecture generated in the first stage, Represents the filtered network architecture, and argmin represents the function used to calculate the minimum value.

[0047] Preferably, when fine-tuning the model, select the candidate network structure set F CR Select the optimal network structure f so that it can be used in the target domain test set D t-test The prediction loss on Minimize, the target domain loss minimization process is expressed as:

[0048]

[0049] in, The optimal parameters of the network structure f obtained through pre-training or previous learning stages. This process evaluates the performance of different network structures on the target domain test data, selects the network architecture that best adapts to the target domain feature distribution, and ensures the model's generalization ability in cross-domain scenarios;

[0050] During the fine-tuning process, the optimization algorithm uses the Adam optimizer, the activation function is set to the Sigmoid function, and the loss function is the binary cross entropy function.

[0051] Preferably, the migration method further comprises:

[0052] A loss weighted optimization strategy is introduced into the obtained target domain model to reduce the correlation between fused features, adjust the ratio between classification loss value and sample weight, and optimize the training process of the fused data model;

[0053] in:

[0054] Random Fourier features RFF are used to map features into a high-dimensional feature space;

[0055] In the weight learning process, the global feature set (F G1 ,F G2 ,...,F Gi ,F L ) and the current batch local features F L Splice to form the comprehensive feature F0 for the next batch optimization; similarly, the global weight set (W G1 ,W G2 ,...,W Gi ,W L ) and the current batch local weight W L Splicing, get the comprehensive weight W0, by iteratively saving and reloading the weight method to reduce storage and computing costs, the feature F0 and weight W0 used to optimize the weight of the next batch of samples in each batch are calculated as follows:

[0056]

[0057]

[0058] Among them, the global feature F Gi and the global weight W Gi Represents the global information accumulated in the historical batches, which remains unchanged during each batch training process. Only the local features F of the current batch are L and local weight W L Trainable updates under the condition of minimizing feature dependencies;

[0059] Then, at the end of each iteration, the smoothing parameter For global information (F Gi ,W Gi ) and the current batch local information (F L ,W L ) to perform weighted fusion and obtain the next round of global information, which is expressed as follows:

[0060]

[0061]

[0062] Where, Used to adjust the memory length of global information: When it approaches 1, the model relies more on historical global information; when When it approaches 0, the model pays more attention to the local information of the current batch; this mechanism achieves a dynamic balance between global knowledge and real-time features, avoiding over-reliance on historical information or local noise;

[0063] Then, the dependence between features is minimized by minimizing the Frobenius norm of the covariance matrix between weighted sample features. The square of the Frobenius norm of the covariance matrix of the weighted features is The expression is as follows:

[0064] in, Represents the weighted mutual information index between feature groups A and B, which is used to measure the strength of the correlation between the two. The smaller the value, the weaker the correlation between the features. is the square of the Frobenius norm, that is, the sum of the squares of the matrix elements;

[0065] Furthermore, the weighted covariance matrix It is expressed as follows:

[0066]

[0067] Where u(·) and v(·) are random Fourier feature RFF mapping functions, which map the original features to high-dimensional space to simplify linear calculations; w i is the sample weight, satisfying The normalization condition of , which is used as an independence test statistic. The smaller its value, the lower the linear correlation between features A and B. Thus, by optimizing the weight w, the model is forced to learn low-correlation fusion features, thereby improving the independence of feature representation;

[0068] Finally, the weight matrix W(*) that achieves optimal feature independence is element-wise multiplied with the model’s classification loss cot(*) to form the final training loss function, expressed as: Loss = cot(*)⊙W(*), where ⊙ represents the element-wise product.

[0069] Compared with the existing technology, the technical advantages of this application are:

[0070] First, the present invention fuses the optimal features of near-infrared spectral data, Raman spectral data, and microscopic image data. The features fused by the self-attention mechanism are then fused through the cross-attention mechanism to form the optimal fused features. This effectively helps in comprehensively analyzing the condition of crops. The complementarity of multi-source data can effectively enhance the generalization ability of the model.

[0071] Second, the present invention further combines semi-supervised active learning, neural network architecture search (NAS) and transfer learning technology, and uses a loss-stabilized weighted strategy to significantly reduce the dependence on a large amount of labeled data during model training, thereby improving the applicability of the model in data-scarce scenarios. It also captures more effective features by optimizing the loss function, thereby reducing the correlation between features, optimizing the model training process, and improving the transfer effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0073] In the attached figure:

[0074] Figure 1 This is a flowchart of an implementation of a network architecture search and synchronous transfer learning method for multimodal graph data according to an embodiment of the present application;

[0075] Figure 2 This is a system architecture diagram for feature fusion in a network architecture search and synchronous transfer learning method for multimodal graph data according to an embodiment of the present application;

[0076] Figure 3 This is a system architecture diagram of a network architecture search in a synchronous transfer learning method for multimodal graph data according to an embodiment of the present application;

[0077] Figure 4 This is a system architecture diagram for loss-stabilized weighting in a synchronous transfer learning method for network architecture search of multimodal graph data according to an embodiment of the present application;

[0078] Figure 5 It is a structural diagram of a network architecture search and synchronous transfer learning system for multimodal graph data according to an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0080] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0081] According to an embodiment of the present application, a method embodiment of a network architecture search synchronous transfer learning method for multimodal graph data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0082] Figure 1 Flowchart of a method for synchronous transfer learning of network architecture search for multimodal graph data according to an embodiment of the present application. Figure 1 As shown, the synchronous transfer learning method includes the following steps:

[0083] S101: Feature extraction and fusion of collected near-infrared spectra, Raman spectra and microscopic images;

[0084] Specifically, the collected near-infrared spectra, Raman spectra, and microscopic images are preprocessed and feature extracted respectively, and the extracted features are fused through the cross-attention mechanism to form a new optimal fusion feature;

[0085] like Figure 1 and Figure 2 As shown, in the above step S101, by fusing multi-source data with complementary information such as near-infrared spectrum, Raman spectrum and microscopic image, the features of the three types of data are optimized and fused, providing effective support for comprehensive analysis of crop status;

[0086] like Figure 1 and Figure 3 As shown, the synchronous transfer learning method provided by the embodiment of the present disclosure also includes the following steps:

[0087] S102: In the constructed SSL_NST module, the fused dataset is divided into source and target domains through transfer learning technology. The neural network architecture search (NAS) adopts a three-stage optimization strategy including network search, fine-tuning, and feedback.

[0088] Specifically, in the network search phase, the default search space of the neural network architecture search (NAS) is replaced with the architecture space combining CNN and RNN, and the neural network architecture search (NAS) is used to search for source domain models on source domain data.

[0089] Specifically, this application replaces the default search space for neural network architecture search (NAS) with a CNN-RNN combined architecture (CR) space. This combines the CNN's ability to extract local and abstract features from the raw spectrum with the RNN's advantage in learning the various dependencies of sequence features. In data fusion tasks, complex model structures may have a higher fitting ability on the fused features, so convolution combined with recurrent neural networks continues to show good classification performance.

[0090] In the fine-tuning phase, a semi-supervised self-training algorithm is used based on the source domain model to assign pseudo-labels to the target domain data. Maximum entropy sampling in active learning is used to filter out pseudo-labeled data, freeze the network structure of the source domain model, and fine-tune the source domain model using the filtered pseudo-labeled data. By unfreezing the network layers layer by layer and adjusting the optimizer and learning rate, the target domain loss is minimized.

[0091] In the feedback phase, the loss of the fine-tuned model on the target domain task is used as a feedback signal to act on the search strategy of the network search and guide the next round of search process. After a preset number of iterations, the searched network structure is used as the target domain model for the target domain task.

[0092] In step S102 of the present disclosure, the SSL_NST module divides the fused dataset into a source domain and a target domain through transfer learning technology. The neural network architecture search (NAS) adopts a three-stage optimization strategy, which mainly includes three stages: network search, fine-tuning, and feedback. A semi-supervised active learning method is introduced into the fine-tuning mechanism to assign pseudo-labels to the target domain data, so that the searched model architecture can better adapt to the target task during the optimization process.

[0093] like Figure 1 and Figure 4 As shown, the synchronous transfer learning method provided by the embodiment of the present disclosure also includes the following steps:

[0094] S103: Introducing a loss weighted optimization strategy into the obtained target domain model to reduce the correlation between fusion features, adjust the ratio between the classification loss value and the sample weight, and optimize the training process of the fusion data model.

[0095] The network architecture search synchronous transfer learning method of this application combines semi-supervised active learning and transfer learning techniques, and then uses a loss-stabilized weighted strategy to transfer knowledge in related fields to the target task, so that the model can still maintain stable learning ability under the condition of a small amount of labeled training data. It also captures more effective features by optimizing the loss function, thereby reducing the correlation between features, optimizing the model training process, and improving the transfer effect. It effectively alleviates the data scarcity problem of asymptomatic samples and realizes the early diagnosis of crop diseases.

[0096] In the above embodiment, the steps of extracting and fusing features of the collected near-infrared spectrum, Raman spectrum and microscopic image include: normalizing the acquired near-infrared spectrum, Raman spectrum and microscopic image using Min-Max standardization and preprocessing them using multivariate scattering correction MSC; extracting 128-dimensional optimal features F-nir and F-raman from the preprocessed near-infrared spectrum and Raman spectrum respectively using a self-attention mechanism; forming a 256-dimensional fusion feature F-nir-raman from the 128-dimensional optimal features of the near-infrared spectrum and Raman spectrum using a splicing method; adjusting the image size of the preprocessed microscopic image to 224×224 pixels and extracting the 256-dimensional optimal feature F-micro image using a self-attention mechanism; and fusing the 256-dimensional fusion feature F-nir-raman and the 256-dimensional optimal feature F-micro image using a cross-attention mechanism to form a 512-dimensional optimal fusion feature F-fusion.

[0097] Furthermore, the steps of extracting the 256-dimensional optimal feature F-microimage using the self-attention mechanism provided in the embodiment of the present disclosure include: setting the input feature sequence When extracting 128-dimensional features from near-infrared spectroscopy and Raman spectroscopy, the learnable weight matrix W is first used. Q 、W K 、W V ∈R d×128 , the input features Mapped to 128-dimensional query vectors , key vector Sum value vector ; Then calculate the dot product of the query vector and all key vectors and divide by the square root of the dimension to scale, the scaling factor is , and then generate the attention weight matrix through the Softmax function, which represents the degree of correlation between the features at each position in the input sequence; finally, multiply the attention weight by the value vector to obtain the 128-dimensional self-attention output feature that integrates the internal dependencies of the sequence , realize contextual semantic modeling of spectral modal features;

[0098] When extracting 256-dimensional features from microscopic images, the learnable weight matrix , , Map the input features into 256-dimensional query, key, and value vectors; use a similar scaled dot product attention calculation process as the spectral modality, but adjust the dimension scaling factor to , and finally generate 256-dimensional self-attention output features , adapt to the high-dimensional feature representation requirements of image modalities, strengthen the long-distance dependency modeling capabilities of image features, and generate 256-dimensional self-attention output features :

[0099] Therefore, the 128-dimensional self-attention output feature and 256-dimensional self-attention output features Respectively expressed as:

[0100]

[0101]

[0102] Where X represents the input feature, represents the weight matrix of the value vector V, represents the weight matrix of the query vector Q, represents the weight matrix of the key vector K, and T represents the transpose of the matrix.

[0103] The steps of using the cross attention mechanism to fuse features to form the 512-dimensional optimal fusion feature F-fusion include:

[0104] set up It is the 256-dimensional fusion feature after the near-infrared spectrum and Raman spectrum are spliced ​​together. The 256-dimensional features extracted from the microscopic image are first passed through the learnable weight matrix 、 、 , spectral fusion features Mapping to query vector , the image features Mapping to key vector Sum value vector ; Then the dot product of the spectral query vector and the image key vector is calculated and scaled, and the cross-modal attention weight is generated through Softmax, which represents the dependency of the spectral feature on the image feature;

[0105] Multiply the attention weight by the image value vector to obtain the response characteristics of the image features to the spectral features;

[0106] Finally, the original spectral fusion feature F3 and the cross-modal response feature are concatenated along the feature dimension through the Concat operation to generate a 512-dimensional cross-modal fusion feature F5, thereby realizing multimodal information interaction and feature fusion of near-infrared spectrum, Raman spectrum and microscopic image. The 512-dimensional fusion feature F5 output by the cross-attention mechanism can be expressed as:

[0107]

[0108] Where F3 represents the spectral fusion feature, represents the learnable weight matrix of the query vector, F4 represents the image features, represents the learnable weight matrix of the key vector, Represents the learnable weight matrix of the value vector, and T represents the transpose of the matrix.

[0109] Optionally, the step of assigning pseudo labels to the target domain data using a semi-supervised self-training algorithm and filtering out pseudo-labeled data using maximum entropy sampling in active learning includes:

[0110] Set the fusion dataset to , where x i represents the sample of the fused dataset, y i Indicates the label corresponding to the sample, n is the number of labeled samples; the source domain dataset is , where m is the number of source domain samples;

[0111] The neural network model generated by the NAS architecture is used as the classification model S. This model fully learns the data distribution P(x, y) of the source domain data (where P(x, y) represents the joint probability distribution) and then uses the confidence threshold mechanism in the self-training algorithm to screen the target domain samples. First, the model C is used to classify the target domain unlabeled sample set D. u Each sample x in j Make predictions and get prediction results and its corresponding prediction confidence(x j ), then set a confidence threshold τ, and filter out samples with confidence higher than the threshold by comparing the prediction confidence of each sample with the threshold, and the filtered target domain sample set By all satisfying confidence(x j )>τ conditional sample x j Composition, that is, only retaining samples with high confidence in the model's prediction results for subsequent tasks to improve the model's performance in the target domain;

[0112] The selected target domain sample set Expressed as:

[0113]

[0114] Where x j represents the sample, D u represents the sample set, τ represents the confidence threshold;

[0115] Then, the information entropy of the selected target domain samples is calculated. The larger the information entropy, the more information the sample contains. In the process of calculating the information entropy, the conditional probability p(y c |x), then take the natural logarithm of the conditional probability of each category, multiply it by the conditional probability and take the negative value to get the uncertainty contribution of a single category; finally, sum the uncertainty contributions corresponding to all categories to get the information entropy H(x) of sample x. The larger the information entropy, the more ambiguous the category of the sample is and the higher the potential information value it contains;

[0116] Among them, the target domain sample x j The information entropy is expressed as:

[0117]

[0118] Where c represents the category of the target domain sample x, c=1,2,...,C, C is the total number of categories; p(y c |x) represents the conditional probability, that is, the posterior probability that sample x belongs to category c under given features;

[0119] Finally, at each iteration In the active learning, the target domain data set D is sampled based on the idea of ​​uncertain maximum entropy sampling. t Sampling is performed, and the sampling function is assumed to be S(D t ), find the most valuable sample subset D through this function v =S(D t ), and these samples are labeled by the source domain model and given pseudo labels .

[0120] Preferably, the step of searching for a source domain model on the source domain data using a neural network architecture search (NAS) comprises:

[0121] In the entire search process, the first stage of the search is to use Gaussian process to search the candidate network architecture f in the source domain data D u The prediction loss on is modeled as:

[0122]

[0123] Where u(·) represents the mean of the Gaussian process prediction loss, reflecting the average performance of the model on the source domain; σ(·) represents the standard deviation of the Gaussian process prediction loss, which is used to characterize the uncertainty of the model performance;

[0124] The balance factor β is used to adjust the degree of attention paid to the mean and uncertainty of model performance:

[0125] When β is large, the search process tends to explore architectures with high uncertainty, which helps to discover potentially better model structures;

[0126] When β is small, the search process focuses more on verified low-loss architectures, accelerating convergence to the local optimal solution;

[0127] Generate candidate architectures in the first stage Based on the minimization of candidate architectures In the target domain data D t The prediction loss on , filter out the network architecture that best suits the target domain This process uses a small amount of labeled data in the target domain to evaluate and select candidate architectures, ensuring that the final model structure can be effectively transferred to the target domain and alleviating the domain differences between the source and target domains. This optimization goal guides the NAS process to search in a direction with stronger cross-domain adaptability by directly linking the target domain performance, thus achieving synchronous and stable migration of the model structure. The optimization goal is expressed as: , where Representing candidate architectures In the target domain data D t The prediction loss on represents the candidate architecture generated in the first stage, Represents the filtered network architecture, and argmin represents the function used to calculate the minimum value.

[0128] Preferably, when fine-tuning the model, select the candidate network structure set F CR Select the optimal network structure f so that it can be used in the target domain test set D t-test The prediction loss on Minimize, the target domain loss minimization process is expressed as:

[0129]

[0130] in, The optimal parameters of the network structure f obtained through pre-training or previous learning stages. This process evaluates the performance of different network structures on the target domain test data, selects the network architecture that best adapts to the target domain feature distribution, and ensures the model's generalization ability in cross-domain scenarios;

[0131] During the fine-tuning process, the optimization algorithm uses the Adam optimizer, the activation function is set to the Sigmoid function, and the loss function is the binary cross entropy function.

[0132] Preferably, the step of introducing a loss weighted optimization strategy into the obtained target domain model to reduce the correlation between fusion features, adjust the ratio between the classification loss value and the sample weight, and optimize the training process of the fusion data model includes:

[0133] Random Fourier features (RFF) are used to map features into a high-dimensional feature space to reduce feature complexity and achieve linear calculation.

[0134] In the weight learning process, the global feature set (F G1 ,F G2 ,...,F Gi ,F L ) and the current batch local features F L Splice to form the comprehensive feature F0 for the next batch optimization; similarly, the global weight set (W G1 ,W G2 ,...,W Gi ,W L ) and the current batch local weight W L Splicing, get the comprehensive weight W0, by iteratively saving and reloading the weight method to reduce storage and computing costs, the feature F0 and weight W0 used to optimize the weight of the next batch of samples in each batch are calculated as follows:

[0135]

[0136]

[0137] Among them, the global feature F Gi and the global weight W Gi Represents the global information accumulated in historical batches. Specifically, F Gi represents the global features accumulated in the i-th group of history, W Gi Represents the corresponding cumulative weight; it remains unchanged during each batch training process, only the local features of the current batch and local weights Trainable updates under the condition of minimizing feature dependencies;

[0138] Then, at the end of each iteration, the smoothing parameter For global information (F Gi ,W Gi ) and the current batch local information (F L ,W L) to perform weighted fusion and obtain the next round of global information, which is expressed as follows:

[0139]

[0140]

[0141] Where, Used to adjust the memory length of global information: When it approaches 1, the model relies more on historical global information; when When it approaches 0, the model pays more attention to the local information of the current batch; this mechanism achieves a dynamic balance between global knowledge and real-time features, avoiding over-reliance on historical information or local noise; represents the global features after weighted fusion, represents the global weight after weighted fusion, F L 、W L Represent the local features and local weights of the current batch, respectively. represents the smoothing parameter, F Gi 、W Gi Represent global features and global weights respectively;

[0142] Then, the dependence between features is minimized by minimizing the Frobenius norm of the covariance matrix between weighted sample features. The square of the Frobenius norm of the covariance matrix of the weighted features is The expression is as follows:

[0143] ;

[0144] in, Represents the weighted mutual information index between feature groups A and B, which is used to measure the strength of the correlation between the two. The smaller the value, the weaker the correlation between the features. is the square of the Frobenius norm, that is, the sum of the squares of the matrix elements;

[0145] Furthermore, the weighted covariance matrix It is expressed as follows:

[0146]

[0147] in, Each element represents the covariance between a certain feature dimension of A and a certain feature dimension of B; n represents the total number of samples, A i and B iThey represent the relevant features and irrelevant features of sample i respectively, T represents the transpose of the matrix or vector, which is used to calculate the correlation between features; u(·) and v(·) are random Fourier feature RFF mapping functions, which map the original features to a high-dimensional space; w i is the sample weight, satisfying The normalization condition weakens the influence of false correlation by adaptively adjusting the contribution of samples to the correlation calculation; and is a weighted mean vector, ensuring that the correlation measure is not disturbed by mean shift;

[0148] This norm is used as an independence test statistic. The smaller its value, the lower the linear correlation between features A and B. Thus, by optimizing the weight w, the model is forced to learn low-correlation fusion features, thereby improving the independence of feature representation.

[0149] Finally, the weight matrix W(*) that achieves optimal feature independence is element-wise multiplied with the model’s classification loss cot(*) to form the final training loss function, expressed as: Loss = cot(*)⊙W(*), where ⊙ represents the element-wise product.

[0150] In the embodiment of the present disclosure, the weight matrix W(*) that achieves the best feature independence is element-wise multiplied by the classification loss cot(*) of the model to form the final training loss function; on the one hand, the constraint of the traditional classification loss on the prediction accuracy is retained, and on the other hand, the feature independence regularization term is introduced through the weight matrix, forcing the model to automatically learn a weight allocation strategy that can minimize feature correlation while reducing the classification error, thereby achieving joint optimization of classification performance and feature quality.

[0151] Figure 5 This is a structural diagram of a network architecture search and synchronous transfer learning system for multimodal graph data according to an embodiment of the present application, such as Figure 5 As shown, the system includes:

[0152] Feature fusion module 201, for extracting and fusing features of collected near-infrared spectra, Raman spectra, and microscopic images;

[0153] The SSL_NST module 202 is used to divide the fused dataset into a source domain and a target domain by using transfer learning technology. The neural network architecture search (NAS) adopts a three-stage optimization strategy including network search, fine-tuning, and feedback.

[0154] The loss stabilization weighting module 203 is used to introduce a loss weighting optimization strategy into the obtained target domain model to reduce the correlation between fusion features, adjust the ratio between the classification loss value and the sample weight, and optimize the training process of the fusion data model.

[0155] Preferably, the SSL_NST module 202 is also used to execute, in the network search phase, replacing the default search space of the neural network architecture search NAS with the CNN and RNN combined architecture space, and using the neural network architecture search NAS to search for the source domain model on the source domain data; in the fine-tuning phase, using the semi-supervised self-training algorithm based on the source domain model to assign pseudo labels to the target domain data, and using the maximum entropy sampling in active learning to filter out the pseudo-label data, freeze the network structure of the source domain model, and use the filtered pseudo-label data to fine-tune the source domain model, and minimize the target domain loss by unfreezing the network layer layer by layer, adjusting the optimizer and the learning rate; in the feedback phase, the loss of the fine-tuned model on the target domain task is used as a feedback signal to act on the search strategy of the network search to guide the next round of search process; after a preset number of iterations, the searched network structure is used as the target domain model for the target domain task.

[0156] It should be noted that the above Figure 5 Each module in the embodiment can be a program module (for example, a set of program instructions that implement a specific function) or a hardware module. For the latter, it can be expressed in the following forms, but is not limited to this: each of the above modules is expressed in the form of a processor, or the functions of each of the above modules are implemented by a processor.

[0157] It should be noted that Figure 5 The preferred implementation of the embodiment shown can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0158] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and exemplary embodiments. They can be applied to a variety of fields suitable for the present invention. Further modifications will be readily apparent to those skilled in the art. Therefore, the present invention is not limited to the specific details and illustrations shown and described herein without departing from the general concept defined by the claims and their equivalents.

Claims

1. A network architecture search and simultaneous transfer learning method for multimodal graph data, characterized by: The following steps are involved: Perform feature extraction and fusion on the collected near-infrared spectra, Raman spectra and microscopic images; In the constructed SSL_NST module, the fused dataset is divided into source and target domains through transfer learning. The neural network architecture search (NAS) adopts a three-stage optimization strategy including network search, fine-tuning, and feedback. In the network search stage, the default search space of the neural network architecture search NAS is replaced with the CNN and RNN combined architecture space, and the neural network architecture search NAS is used to search for the source domain model on the source domain data; in the fine-tuning stage, a semi-supervised self-training algorithm is used based on the source domain model to assign pseudo labels to the target domain data, and the maximum entropy sampling in active learning is used to screen out the pseudo-labeled data. Among them, the neural network model generated by the NAS architecture is used as the classification model S. Model S fully learns the data distribution P(x, y) of the source domain data and uses the confidence threshold mechanism in the self-training algorithm to screen the target domain samples; first, the model S is used to classify the target domain unlabeled sample set D u Each sample x in j Make predictions and get prediction results and its corresponding prediction confidence(x j ); Then set a confidence threshold τ, and compare the prediction confidence of each sample with the threshold to filter out samples with confidence higher than the threshold. The filtered target domain sample set D t By all satisfying confidence(x j )>τ conditional sample x j The target domain sample set is filtered out; then, the information entropy of the filtered target domain samples is calculated. In the process of calculating the information entropy, the conditional probability p(y c ∣x), then take the natural logarithm of the conditional probability of each category, multiply it by the conditional probability and take the negative value to get the uncertainty contribution of a single category; finally, sum the uncertainty contributions corresponding to all categories to get the information entropy H(x) of sample x; in each iteration t, combine the idea of ​​maximum entropy sampling based on uncertainty in active learning to sample the target domain data set D t Sampling is performed based on the sampling function S(D t ) Get the required sample subset D v =S(D t ), and hand these samples over to the source domain model for annotation and assign them pseudo labels; The network structure of the source domain model is frozen, and the source domain model is fine-tuned using the filtered pseudo-label data. The target domain loss is minimized by unfreezing the network layers layer by layer and adjusting the optimizer and learning rate. In the feedback phase, the loss of the fine-tuned model on the target domain task is used as a feedback signal to act on the search strategy of the network search and guide the next round of search process. After a preset number of iterations, the searched network structure is used as the target domain model for the target domain task.

2. The method for network architecture search and synchronous transfer learning for multimodal graph data according to claim 1, characterized in that: The steps of feature extraction and fusion of the collected near-infrared spectra, Raman spectra and microscopic images include: The acquired near-infrared spectra, Raman spectra and microscopic images were normalized using Min-Max normalization and pre-processed using multivariate scattering correction (MSC); The preprocessed near-infrared spectrum and Raman spectrum are respectively extracted with the self-attention mechanism to obtain the 128-dimensional optimal features F-nir and F-raman; The 128-dimensional optimal features of near-infrared spectroscopy and Raman spectroscopy are concatenated to form a 256-dimensional fusion feature F-nir-raman; The image size of the preprocessed microscopic image is resized to 224×224 pixels and the self-attention mechanism is used to extract the 256-dimensional optimal feature F-micro image; The 256-dimensional fusion feature F-nir-raman and the 256-dimensional optimal feature F-micro image are fused using the cross attention mechanism to form a 512-dimensional optimal fusion feature F-fusion.

3. The network architecture search and synchronous transfer learning method for multimodal graph data according to claim 2 is characterized in that: The step of extracting the 256-dimensional optimal feature F-micro image using the self-attention mechanism includes: Set the input feature sequence X∈R n×d When extracting 128-dimensional features from near-infrared spectroscopy and Raman spectroscopy, the learnable weight matrix W is first used. Q 、W K 、W V ∈R d×128 , map the input features X to 128-dimensional query vectors Q=XW Q , key vector K=XW K Sum vector V=XW V ; Then calculate the dot product of the query vector and all key vectors and divide by the square root of the dimension to scale, the scaling factor is , and then generate the attention weight matrix through the Softmax function, which represents the degree of correlation between the features at each position in the input sequence; finally, multiply the attention weight by the value vector to obtain the 128-dimensional self-attention output feature that integrates the internal dependencies of the sequence ; When extracting 256-dimensional features from microscopic images, the input features are mapped into 256-dimensional query, key, and value vectors; the dimension scaling factor is adjusted to , generating 256-dimensional self-attention output features ; 128-dimensional self-attention output features and 256-dimensional self-attention output features Respectively expressed as: ； ； Where X represents the input feature, W V Represents the weight matrix of the value vector V, W Q Represents the weight matrix of the query vector Q, W K represents the weight matrix of the key vector K, and T represents the transpose of the matrix.

4. The method for network architecture search and synchronous transfer learning for multimodal graph data according to claim 3, characterized in that: The step of using the cross attention mechanism to perform feature fusion to form a 512-dimensional optimal fusion feature F-fusion includes: set up It is the 256-dimensional fusion feature after the near-infrared spectrum and Raman spectrum are spliced ​​together. The 256-dimensional features extracted from the microscopic image are first passed through the learnable weight matrix , map the spectral fusion feature F3 to the query vector , mapping the image feature F4 to a key vector Sum value vector ; Then the dot product of the spectral query vector and the image key vector is calculated and scaled, and the cross-modal attention weight is generated through Softmax; Multiply the attention weight by the image value vector to obtain the response characteristics of the image features to the spectral features; The original spectral fusion feature F3 and the cross-modal response feature are concatenated along the feature dimension through the Concat operation to generate a 512-dimensional cross-modal fusion feature F5. The 512-dimensional fusion feature F5 output by the cross-attention mechanism can be expressed as: ； Where F3 represents the spectral fusion feature, represents the learnable weight matrix of the query vector, F4 represents the image features, represents the learnable weight matrix of the key vector, Represents the learnable weight matrix of the value vector, and T represents the transpose of the matrix.

5. The method for network architecture search and synchronous transfer learning for multimodal graph data according to claim 4, characterized in that: In the step of filtering out pseudo-label data, the fusion dataset is set to , where x i represents the sample of the fused dataset, y i Indicates the label corresponding to the sample, n is the number of labeled samples; The source domain dataset is , where m is the number of source domain samples, x j Represents samples of the source domain dataset; Target domain sample set D t Expressed as: ； Where x j represents the sample, D u represents the sample set, τ represents the confidence threshold; Target domain sample x j The information entropy is expressed as: ； Where c represents the category of the target domain sample x, c=1,2,...,C, C is the total number of categories; p(y c |x) represents the conditional probability, that is, the posterior probability that sample x belongs to category c under given features.

6. The method for network architecture search and synchronous transfer learning for multimodal graph data according to claim 5, characterized in that: The step of searching for a source domain model on source domain data using a neural network architecture search (NAS) includes: In the entire search process, the first stage of the search is to use Gaussian process to search the candidate network architecture in the source domain data D u The prediction loss on is modeled as: ； Where u(·) represents the mean of the Gaussian process prediction loss, σ(·) represents the standard deviation of the Gaussian process prediction loss, β represents the balance factor, and D u represents the source domain dataset, Cost(f, D u ) represents the network architecture f in the source domain data D u Prediction loss on ; The degree of attention paid to the mean and uncertainty of model performance is adjusted by the balance factor β; Generate candidate architectures in the first stage Based on the minimization of candidate architectures In the target domain data D t The prediction loss on , filter out the network architecture that best suits the target domain , the optimization objective is expressed as: ； Where, Representing candidate architectures In the target domain data D t The prediction loss on represents the candidate architecture generated in the first stage, Represents the filtered network architecture, and argmin represents the function used to calculate the minimum value.

7. The method for network architecture search and synchronous transfer learning for multimodal graph data according to claim 6, characterized in that: When fine-tuning the model, the optimal network structure f is selected from the candidate network structure set to minimize its prediction loss on the target domain test set. The target domain loss minimization process is expressed as: ； in, represents the prediction loss, is the optimal parameter of the network structure f obtained through pre-training or previous stage learning, F CR represents the set of candidate network structures, D t-test represents the target domain test set; During the fine-tuning process, the optimization algorithm uses the Adam optimizer, the activation function is set to the Sigmoid function, and the loss function is the binary cross entropy function.

8. The method for network architecture search and synchronous transfer learning for multimodal graph data according to claim 7, characterized in that: The steps of the migration method also include: A loss weighted optimization strategy is introduced into the obtained target domain model to reduce the correlation between fused features, adjust the ratio between classification loss value and sample weight, and optimize the training process of the fused data model; in: Random Fourier features RFF are used to map features into a high-dimensional feature space; In the weight learning process, the global feature set (F G1 ,F G2 ,...,F Gi ,F L ) and the current batch local features F L Splicing is performed to form a comprehensive feature F0 for the next batch optimization; Similarly, the global weight set (W G1 ,W G2 ,...,W Gi ,W L ) and the current batch local weight W L Splicing, we get the comprehensive weight W0. The calculation formula of the feature F0 and weight W0 used to optimize the weight of the next batch of samples in each batch is as follows: ； ； Among them, the global feature F Gi and the global weight W Gi Represents the global information accumulated in the historical batches, which remains unchanged during each batch training process. Only the local features F of the current batch are L and local weight W L Trainable updates under the condition of minimizing feature dependencies; At the end of each iteration, the smoothing parameter For global information (F Gi ,W Gi ) and the current batch local information (F L ,W L ) to perform weighted fusion and obtain the next round of global information, which is expressed as follows: ； ； Where, Used to adjust the memory length of global information, represents the global features after weighted fusion, represents the global weight after weighted fusion, F L 、W L Represent the local features and local weights of the current batch, respectively. represents the smoothing parameter, F Gi 、W Gi Represent global features and global weights respectively; Minimize the Frobenius norm of the cross-covariance matrix between weighted sample features and the square of the Frobenius norm of the covariance matrix of the weighted features The expression is as follows: ； in, Represents the weighted mutual information index between feature groups A and B, which is used to measure the strength of the correlation between the two; is the square of the Frobenius norm; Weighted covariance matrix It is expressed as follows: ； in, Each element represents the covariance between a certain feature dimension of A and a certain feature dimension of B; n represents the total number of samples, A i and B i They represent the relevant features and irrelevant features of sample i respectively, T represents the transpose of the matrix or vector, which is used to calculate the correlation between features; u(·) and v(·) are random Fourier feature RFF mapping functions, which map the original features to a high-dimensional space; w i is the sample weight, satisfying Normalization conditions of ; and is the weighted mean vector; The weight matrix W(*) that achieves optimal feature independence is element-wise multiplied with the model's classification loss cot(*) to form the final training loss function, expressed as: Loss=cot(*)⊙W(*), where ⊙ represents the element-wise product.

Images