Multi-modal fusion signal classification method based on evolutionary neural architecture search
By employing a multimodal fusion method that combines multimodal feature extraction and evolutionary neural architecture search, the problem of instability in single-modal feature extraction and fusion strategies in signal classification is solved, achieving efficient and adaptive signal classification and improving classification accuracy and robustness.
Patent Information
- Application Number
- CN202511816477.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-03-06
AI Technical Summary
In existing signal classification methods, single-modal feature extraction methods cannot fully capture signal information, and multimodal fusion strategies lack adaptability and universality, and have low search efficiency, which limits classification performance.
We employ multimodal feature extraction models (AWN, MCLDNN, ResNet, LSTM, GRU) combined with evolutionary neural architecture search and directed graph convolutional network (DGCN) for feature fusion, automatically optimize the fusion structure, and achieve adaptive multimodal fusion through binary tree encoding and a set of fusion operators.
It significantly improves the accuracy and robustness of signal classification, avoids information redundancy, adapts to different signal scenarios, improves search efficiency, enhances generalization ability, and is suitable for complex signal environments.
Smart Images
Figure CN121614945A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of signal classification technology, specifically relating to a multimodal fusion signal classification method based on evolutionary neural architecture search. Background Technology
[0002] Signal classification is an important research area in pattern recognition and machine learning, with wide applications in wireless communication, radar signal analysis, and the Internet of Things (IoT). For example, in wireless communication, accurate classification of modulated signals is crucial for the design of intelligent receivers; in radar systems, the classification of target echo signals aids in target identification and threat assessment. However, due to the non-stationarity and high dimensionality of signal data, and the frequent noise interference in practical applications, effectively extracting key signal features and achieving efficient classification remains a significant research challenge.
[0003] Traditional signal classification methods typically rely on manual feature extraction and classical machine learning algorithms. For example, time-domain features (such as mean and variance) and frequency-domain features (such as Fourier transform, short-time Fourier transform (STFT), and wavelet transform) are often used to describe signal characteristics, combined with classifiers such as support vector machines (SVM) and random forests (RF) for decision-making. However, these methods heavily depend on expert experience for manual feature design and selection, which is not only cumbersome but also results in features extracted that are often specific to particular scenarios, leading to limited generalization ability of the model and difficulty in adapting to signal variations in complex environments. Furthermore, different types of signals may have different time-domain characteristics, and a single feature extraction method often cannot comprehensively capture all the information of the signal.
[0004] In recent years, the rise of deep learning technology has provided new solutions for signal classification tasks. Models such as deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and transformers have been widely applied to signal classification. Compared to traditional methods, these deep learning models can automatically learn deep-level features of signals, reducing the need for manual feature engineering and exhibiting higher robustness in complex signal classification tasks. However, despite the significant progress made in deep learning methods, existing approaches still face some challenges.
[0005] Most current deep learning methods typically rely on single-modal feature extraction, extracting signal features from a specific representation space (such as the time domain, frequency domain, or time-frequency domain). This single-modal approach may not fully capture all the information of the signal. For example, relying solely on time-domain features may fail to characterize the signal's spectral distribution, while relying solely on frequency-domain features may ignore the signal's instantaneous dynamic changes. Therefore, multimodal feature extraction and fusion have become an important direction for improving signal classification performance. Existing multimodal fusion methods mostly employ simple feature concatenation or weighted averaging, but these methods often fail to fully utilize the complementary information between different features and may even introduce redundant information, affecting classification performance. Furthermore, while some fusion methods based on attention mechanisms or graph neural networks (GNNs) can enhance the interactivity between features from different modalities, they usually rely on hand-designed fusion strategies, lacking universality and adaptability, making it difficult to determine the optimal fusion method. Summary of the Invention
[0006] The purpose of this invention is to provide a multimodal fusion signal classification method based on evolutionary neural architecture search, which can solve the problems of single-modal feature extraction limitations, instability of feature fusion strategies, difficulty in adapting manually set fusion rules to different task requirements, and low search efficiency in the current field of signal classification.
[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0008] A multimodal fusion method based on evolutionary neural architecture search includes the following steps:
[0009] Step 1: Multimodal Feature Extraction
[0010] The raw wireless signal data is input into the multimodal feature extraction module, which includes an adaptive wavelet network (AWN), a multi-channel convolutional deep neural network (MCLDNN), a residual neural network (ResNet), a long short-term memory network (LSTM), and a gated recurrent unit network (GRU), to obtain mode 1, mode 2, mode 3, mode 4, and mode 5, respectively.
[0011] The modalities 1 to 5 are combined into a modal set, which serves as the basic input data for subsequent fusion architecture search;
[0012] Step 2: Feature Fusion Method Based on Evolutionary Neural Architecture Search
[0013] A set of fusion operators is pre-constructed, which includes: addition fusion operator, multiplication fusion operator, concatenation fusion operator, maximum value fusion operator, and average value fusion operator. The fusion operators are used to perform feature fusion operations on two input modal features or intermediate fusion features.
[0014] A binary tree structure is used to encode the fusion network structure. The leaf nodes of the binary tree are used to represent modes, and the non-leaf nodes are used to represent fusion operators. The maximum depth of the binary tree is 15. The population size is set to 32. According to the encoding rules, 32 individuals are randomly generated to form the initial population, i.e. the initial parent population.
[0015] Decode each individual in the parent population, construct a corresponding multimodal fusion neural network based on the binary tree structure corresponding to each individual, input the constructed fusion neural network into the training set data for real training, use the Adam optimizer to update parameters during the training process, test the trained fusion network on the validation set or test set, and obtain the fitness value corresponding to the fusion method.
[0016] Individuals with obtained true fitness values are transformed into graph structures to construct corresponding graph data. The graph data is then input into the Graph Convolutional Network Performance Predictor (DGCN) for training, enabling the DGCN to learn the mapping relationship between the fusion method and its corresponding fitness value, thus completing the initial training of the performance predictor.
[0017] Based on the fitness value of each individual in the parent population, several parent individuals are selected according to the binary tournament strategy. The selected parent individuals are cross-crossed in pairs to generate multiple offspring individuals. Then, mutation operations are performed on these individuals. The mutation operations include: modal node replacement, fusion operator node replacement, subtree structure replacement or pruning, thus completing the generation of a set of offspring individuals.
[0018] Each offspring fusion structure individual is converted into corresponding graph data, and the graph data is input into the pre-trained DGCN performance predictor to obtain the predicted fitness value of each offspring individual. All offspring individuals are then ranked according to the predicted fitness value.
[0019] From the offspring individuals produced by each pair of parents, the two offspring individuals with the highest predicted fitness values are selected and added to the offspring population. This process is repeated until the number of offspring individuals reaches the set population size of 32. Then, each individual in the offspring population undergoes a complete evaluation to obtain the true fitness value of each individual in the offspring population. The new true fitness value and its corresponding fusion method are then added to the predictor training dataset to incrementally train the DGCN performance predictor, thereby updating the performance predictor parameters and improving the prediction accuracy generation by generation.
[0020] The previous generation of parent populations was merged with the offspring populations, and a selection strategy was used to select 32 individuals as the new generation of parent populations based on the true fitness value of each individual.
[0021] Determine if the preset termination condition is met: reaching the maximum number of iterations or no significant improvement in the optimal fitness over multiple generations. If the termination condition is not met, return to the steps after forming the parent population; if the termination condition is met, select the individual with the highest fitness value as the optimal individual. After decoding, the optimal fusion method is formed. Based on the optimal fusion method, construct the final network model and complete the final training.
[0022] Furthermore, the graph convolutional network performance predictor DGCN is a directed graph convolutional network, and its graph data construction process includes: taking each node in the binary tree structure as a graph node, establishing directed edges according to the parent-child connection relationship in the binary tree, thereby converting the individual binary tree structure into a directed graph.
[0023] Furthermore, the Directed Graph Convolutional Network (DGCN) processes the directed graph by establishing forward propagation paths and backward propagation paths, respectively. Forward propagation proceeds along the direction of the directed edge, while backward propagation proceeds in the opposite direction of the directed edge. The final node features are represented as the average of the forward propagation features and the backward propagation features.
[0024] Furthermore, the loss function used for the initial training and incremental training of the performance predictor is the mean squared error loss function.
[0025] Furthermore, the phrase "selecting the two offspring individuals with the highest predicted fitness values from each pair of parent offspring individuals to add to the offspring population" specifically means: for each pair of parent offspring individuals participating in the crossover operation, only the individuals with the highest and second-highest predicted fitness values are selected from all the offspring individuals they generate to add to the offspring population.
[0026] Furthermore, the process of decoding the fusion structure into a multimodal fusion neural network structure includes: firstly, processing each modal feature through a linear layer, a batch normalization layer, and an activation function layer to obtain the processed modal features; then, according to the hierarchical order indicated by the binary tree structure, sequentially using the fusion operators corresponding to the non-leaf nodes to fuse the leaf node features until the root node, to obtain the final fused features; finally, passing the final fused features through a classification layer to obtain the classification result.
[0027] Furthermore, the fitness value is the signal classification accuracy of the multimodal fusion neural network structure on the validation set or test set.
[0028] Compared with the prior art, the present invention has the following advantages:
[0029] I. Multimodal fusion improves classification performance
[0030] Multiple deep learning models (AWN, MCLDNN, ResNet, LSTM, GRU) are used to extract signal features from different perspectives, forming complementary multimodal representations. This effectively captures multi-dimensional information of the signal, including time domain, frequency domain, and time series, avoiding the limitations of single-modal feature extraction and significantly improving the accuracy and robustness of classification.
[0031] II. Automated Convergence Architecture Search
[0032] The introduction of Evolutionary Neural Architecture Search (ENAS) automatically optimizes the multimodal fusion structure, eliminating the need for manual design of fusion strategies. This avoids the information redundancy or loss problems caused by simple splicing or weighted averaging in traditional multimodal fusion methods, and achieves intelligent and adaptive fusion methods.
[0033] III. High-efficiency search and computational resource optimization
[0034] By combining Directed Graph Convolutional Networks (DGCN) to build a performance predictor, the fitness of candidate fusion structures can be quickly evaluated, avoiding time-consuming and resource-intensive real training for each structure, significantly improving search efficiency, and making it suitable for deployment environments with limited computing resources.
[0035] IV. Strong scalability and versatility
[0036] The fusion of operator sets (such as addition, multiplication, concatenation, maximum value, average value, etc.) and binary tree coding structure is flexible and scalable, suitable for different types of signal classification tasks, and has good task adaptability and scalability.
[0037] V. Integration Strategy: Self-Learning and Continuous Optimization
[0038] The performance predictor performs incremental learning during the evolution process, continuously updating and optimizing as the search progresses, improving prediction accuracy, and gradually approaching the optimal fusion structure, thus achieving continuous self-optimization of the fusion strategy.
[0039] VI. Avoid overfitting and improve generalization ability
[0040] Training strategies such as early stopping, cross-entropy loss function, and Adam optimizer are adopted, and fitness is evaluated using a validation set to effectively control model complexity, avoid overfitting, and enhance the model's generalization ability in real signal environments.
[0041] VII. Suitable for complex signal environments
[0042] The method is validated in real wireless signal datasets (such as RML2016.10a) containing various modulation schemes and signal-to-noise ratios, demonstrating its applicability to real-world signal classification scenarios that are non-stationary, high-noise, and highly variable.
[0043] 8. Clear structure and strong implementability
[0044] The methodology is clearly defined, with clear and operable steps from feature extraction, encoding and decoding, performance prediction to evolutionary search, making it easy to implement in engineering and integrate into systems. Attached Figure Description
[0045] Figure 1 This is a flowchart of the present invention. Detailed Implementation
[0046] The present invention will be further illustrated below with specific examples.
[0047] This embodiment describes a multimodal fusion signal classification method based on evolutionary neural architecture search, which includes the following steps:
[0048] S1. Extract modal features
[0049] Specifically, the RML2016.10a dataset was selected. This is a wireless signal identification dataset containing 11 modulation schemes and 20 signal-to-noise ratios (SNRs). Each modulation scheme and SNR combination contains 2000 signal samples, totaling 220,000 signal samples. Each sample is IQ sampling data with a sampling rate of 2.4 MHz, a sampling depth of 16 bits, and a duration of 128 microseconds. Each sample includes its modulation scheme and SNR. The data was divided into training and test sets in a 6:4 ratio.
[0050] S11, Model Training:
[0051] Each deep learning model (AWN, MCLDNN, ResNet, LSTM, GRU) is trained separately. Let Y be the label for each data point; the training objective is to learn a mapping function through the neural network. Make the predicted output Close to real labels .
[0052] The training process for each model is as follows:
[0053] 1. Define the model structure:
[0054] (1) LSTM: Utilizes a long short-term memory network to capture the temporal dependencies of signals;
[0055] (2) GRU: Uses gated recurrent units to capture long-term dependencies while reducing computational overhead;
[0056] (3) MCLDNN: Effectively captures multi-dimensional information of signals using a spatiotemporal multi-channel learning framework;
[0057] (4) ResNet: Uses residual fast extraction to extract key features in time-series signals;
[0058] (5)AWN: By integrating adaptive wavelet decomposition and channel attention mechanism, multi-band features are clearly extracted and optimized.
[0059] 2. Loss function:
[0060] All model training processes use cross-entropy loss:
[0061] Where C is the number of categories, For real category labels, This represents the predicted probability output by the model.
[0062] 3. Optimizer:
[0063] All model training processes employed the Adam optimization algorithm with a learning rate of 0.001 and exponential decay.
[0064]
[0065] in, Let be the learning rate in round t. The initial learning rate, This is the attenuation coefficient.
[0066] 4. Model training process: Input the original data into the model, perform forward propagation, and calculate the model output. After calculating the error based on the loss function, gradient descent is used to optimize the parameters. The model is trained iteratively for 100 epochs, and an early stopping mechanism is used to prevent overfitting.
[0067] S12: Feature Extraction
[0068] After training, we extract deep features using the saved model weights. Specifically, for each input data... We use the penultimate layer of the model (i.e., the fully connected layer or pooling layer before the classification layer) as its feature representation:
[0069] in, This indicates the portion of the trained model after removing the classification header. These are the extracted features. For each model, we obtain the corresponding feature set:
[0070]
[0071] Finally, a complete set of multimodal features is obtained, which serves as the input for subsequent fusion and optimization.
[0072] S2: Multimodal Evolutionary Fusion
[0073] Since different modalities may contain complementary information, direct splicing or simple weighting may not fully utilize the role of different features. Therefore, this invention adopts the method of Evolutionary Neural Architecture Search (ENAS) combined with a performance predictor to find the optimal modality combination and fusion method.
[0074] S21: Encoding and Decoding of Individuals:
[0075] 1. Encoding: The encoding strategy is a variable-length encoding strategy for deep fusion networks. It consists of two parts: modalities and fusion operators. Let... For modal combination, As a combination of fusion operators, each individual can be represented as:
[0076] in, ,in yes One of the elements; ,in yes One of the elements; each individual's The values may differ.
[0077] 2. Decoding: Each individual is decoded into a deep fusion network. Specifically, let's assume the individual is... Each mode After processing through a linear layer, a BN layer, and a ReLU layer, the modal combination is: ,Will and according to The methods are integrated, and then the integrated result is passed through... and The final characteristics are obtained through fusion. Finally, through a normalization layer and a linearization layer, Convert to probability vector .
[0078] Different fusion strategies can lead to different classification results, so it is necessary to automatically search for the optimal solution.
[0079] S22: Performance predictor based on directed graph convolutional network (DGCN):
[0080] 1. DGCN performance predictor construction:
[0081] Since each individual node is represented as a binary tree structure, DGCN, as an efficient network for processing graph data, can capture the dependencies between nodes in the tree structure. Therefore, DGCN is chosen to build the performance predictor.
[0082] Let individuals , where the set of nodes Representing modalities and fusion operators, edge sets This represents their connection relationship. DGCN transmits information through the adjacency matrix A, and the feature updates for each layer are as follows:
[0083] in: It is the first Layer node characteristics, The trainable weight matrix; It is the adjacency matrix after adding self-loops. It is the corresponding degree matrix; It is the ReLU activation function.
[0084] Traditional graph convolutional networks only consider undirected graphs. This invention designs a directed GCN (Directed GCN) to establish both forward and backward propagation paths, namely:
[0085]
[0086] The final feature is represented as the average of the two:
[0087] 2. DGCN predictor training:
[0088] The network is trained using individuals from the initial population and the actual fitness values of each individual after training as the initial training set. The goal is for the network to be able to predict individual fitness values.
[0089] The loss function uses mean squared error (MSE):
[0090] Gradient descent optimization using the Adam optimizer makes the DGCN predictions closer to the true fitness values.
[0091] 3. DGCN predictor update:
[0092] During the continuous evolution process, as individuals with high fitness values predicted by the predictor are trained, the true fitness values of these individuals are continuously generated. Therefore, after each round of evolution, the DGCN predictor will undergo incremental training to improve its accuracy.
[0093] S23: Evolutionary search process:
[0094] 1. Population initialization:
[0095] Randomly generate N initial individuals and calculate their true fitness values;
[0096] At the same time, these individuals and their true fitness values are used to train the performance predictor.
[0097] 2. Producing offspring:
[0098] Two distinct individuals are selected from the parent population using a binary tournament to generate M offspring.
[0099] Then, these offspring are subjected to crossover and mutation operations to obtain M individuals.
[0100] 3. Fitness assessment:
[0101] After encoding M individuals, a performance predictor is used to quickly estimate their fitness, yielding predicted fitness values. The two best overheating samples from the prediction results are selected for training to obtain their true fitness values.
[0102] 4. Evolutionary process:
[0103] The performance predictor is updated with the latest trained offspring population data to optimize its predictive ability; the two individuals with the highest fitness values among the two individuals in the parent generation and the two trained offspring individuals are selected to enter the next generation until the offspring population is consistent with the initial population, and then the next round of evolution begins; when no better individuals can be found or the maximum number of generations is reached, the individual with the highest fitness value is finally found, which is the optimal structure.
Claims
1. A multi-modal fusion method based on evolutionary neural architecture search, characterized in that, The method comprises the following steps: Step 1: multi-modal feature extraction The original wireless signal data is input into a multi-modal feature extraction module including an adaptive wavelet network (AWN), a multi-channel convolutional deep neural network (MCLDNN), a residual neural network (ResNet), a long short-term memory network (LSTM) and a gated recurrent unit network (GRU), to obtain modalities 1, 2, 3, 4 and 5, respectively; The modalities 1-5 form a modality set as the basic input data for subsequent fusion architecture search; Step 2: feature fusion method based on evolutionary neural architecture search A fusion operator set is constructed in advance, including an addition fusion operator, a multiplication fusion operator, a concatenation fusion operator, a maximum value fusion operator and an average value fusion operator, which are used for feature fusion operation on two input modalities or intermediate fusion features; A binary tree structure is used to encode the fusion network structure, with leaf nodes of the binary tree representing modalities and non-leaf nodes representing fusion operators, and the maximum depth of the binary tree being 15; the population size is set to 32, and 32 individuals are randomly generated according to the encoding rule to form an initial population, i.e., an initial parent population; Each individual in the parent population is decoded, and a corresponding multi-modal fusion neural network is constructed according to the binary tree structure of each individual; the constructed fusion neural network is input into the training set data for real training, and the training process uses an Adam optimizer for parameter updating; the trained fusion network is tested on a validation set or a test set to obtain the fitness value corresponding to the fusion method; The individuals with obtained real fitness values are converted into graph structure to construct corresponding graph data, which are input into a graph convolution network performance predictor (DGCN) for training, so that the DGCN learns the mapping relationship between the fusion method and its corresponding fitness value, and completes the initial training of the performance predictor; According to the fitness values of each individual in the parent population, a number of parent individuals are selected according to a binary tournament strategy, and the selected parent individuals are crossed with each other to generate a plurality of offspring individuals, and then the individuals are subjected to mutation operation, which includes modality node replacement, fusion operator node replacement, sub-tree structure replacement or cutting, i.e., a group of offspring individuals are generated; Each offspring fusion structure individual is converted into corresponding graph data, which are input into the trained DGCN performance predictor to obtain the predicted fitness value corresponding to each offspring individual, and all offspring individuals are arranged according to the predicted fitness values. Two offspring individuals with the highest predicted fitness values are selected from offspring individuals generated from each pair of parents to join the offspring population, and the iteration is continuously performed until the number of offspring population individuals reaches 32 of the set population size, then each individual in the offspring population is completely evaluated to obtain the real fitness value of each individual in the offspring population, and the obtained new real fitness value is added to the predictor training data set together with the corresponding fusion mode, and incremental training is performed in the DGCN performance predictor, so that the performance predictor parameters are updated, and the prediction accuracy is improved generation by generation; The last generation of parent population and offspring population are merged, and each individual in the population is selected according to the real fitness value by using the selection strategy, and 32 individuals are selected as the new generation of parent population; It is judged whether the preset termination condition is met: the maximum number of iterations is reached or the optimal fitness of continuous generations has no obvious improvement, if the termination condition is not met, the step of forming the parent population is returned to be executed; if the termination condition is met, the individual with the highest fitness value is selected as the optimal individual, and the optimal fusion mode is formed after decoding, and the final network model is constructed based on the optimal fusion mode, and the final training is completed.
2. The method of claim 1, wherein, The directed graph convolution network performance predictor DGCN is a directed graph convolution network, and the graph data construction process includes: regarding each node in the binary tree structure as a graph node, and establishing a directed edge according to the parent-child connection relationship in the binary tree, so as to convert the binary tree structure of the individual into a directed graph.
3. The method of claim 2, wherein, The directed graph convolution network DGCN processes the directed graph by establishing a forward propagation path and a reverse propagation path respectively, wherein the forward propagation is performed in the direction of the directed edge, and the reverse propagation is performed in the reverse direction of the directed edge; and the final node feature is represented as the average value of the forward propagation feature and the reverse propagation feature.
4. The method of claim 1, wherein, The loss function used in the initial training and incremental training of the performance predictor is the mean square error loss function.
5. The method of claim 1, wherein, The "selecting two offspring individuals with the highest predicted fitness values from offspring individuals generated from each pair of parents to join the offspring population" specifically includes: for each pair of parent individuals participating in the crossover operation, only the top two individuals in the predicted fitness value are selected from all offspring individuals generated therefrom to join the offspring population.
6. The method of claim 1, wherein, The decoding process of the fusion structure individual into the multi-modal fusion neural network structure includes: first, each modal feature is processed through a linear layer, a batch normalization layer and an activation function layer to obtain a processed modal feature; then, according to the hierarchical order indicated by the binary tree structure, the leaf node features are fused in turn using the fusion operators corresponding to the non-leaf nodes until the root node, and the final fusion feature is obtained; finally, the final fusion feature is classified through a classification layer to obtain a classification result.
7. The method of claim 1, wherein, The fitness value is the signal classification accuracy of the multi-modal fusion neural network structure on the validation set or the test set.