Multi-modal automatic modeling and fusion method and device based on neural architecture search

Abstract

The application provides a multi-modal automatic modeling and fusion method and device based on neural architecture search, and relates to the technical field of computer science. The method comprises the following steps: acquiring multi-modal data and a task type; automatically generating a corresponding optimal unit architecture for each modal data based on neural architecture search, and performing feature extraction on each modal data respectively; generating a dynamically updated correlation heat map according to the inter-modal dependency relationship in the feature level of each modal data, obtaining a specific fusion strategy according to a fusion strategy decision maker, and constructing an optimal fusion network architecture; automatically adjusting an output layer and a loss function according to the task type, obtaining a network suitable for task requirements, and completing different downstream tasks. The application can realize end-to-end joint optimization of single-modal feature extraction and multi-modal fusion strategy, significantly reduce the cost of manual intervention and GPU resource consumption, and improve the task performance and model generalization ability in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer science and technology, and in particular to a method and apparatus for multimodal automatic modeling and fusion based on neural architecture search. Background Technology

[0002] Most existing multimodal learning methods employ manually designed feature extraction networks and fixed fusion strategies. These traditional methods face numerous bottlenecks that are difficult to overcome in practical applications.

[0003] In the field of neural architecture search, Neural Architecture Search (NAS), as a technique for automatically optimizing network architectures, has achieved certain results in unimodal learning. However, existing neural architecture search methods still face many challenges and problems when applied to multimodal feature fusion. First, the computational cost is extremely high. Traditional neural architecture search methods typically require a large amount of computing resources for large-scale searches to explore the optimal network architecture. Some NAS methods may require thousands of GPU hours to complete the search process, which is unaffordable for most researchers and practical applications, greatly limiting their widespread application in the field of multimodal learning. Second, existing NAS methods often focus on optimizing unimodal feature extraction networks or optimizing multimodal fusion strategies as independent modules, making it difficult to achieve joint optimization of unimodal feature extraction and multimodal fusion strategies. This separate optimization approach cannot consider the characteristics of multimodal data and the interrelationship between feature extraction and fusion from a global perspective, thus making it difficult to achieve optimal multimodal modeling results and fully leverage the overall advantages of multimodal data.

[0004] In feature extraction, existing traditional multimodal learning methods largely rely on manually designed network architectures. For example, for image data, predefined CNN (Convolutional Neural Network) architectures are typically used to extract features; for text data, RNN (Recurrent Neural Network) and its variants (such as LSTM and GRU) are often used to process sequence information. However, these manually designed feature extraction networks are highly dependent on the experience and knowledge of domain experts. For new data types or specific tasks, the network architecture needs to be redesigned and adjusted, which is not only time-consuming and labor-intensive, but also makes it difficult to guarantee that the optimal feature extraction network can be designed every time. In addition, fixed manually designed networks are difficult to flexibly optimize for the characteristics of different modalities of data, resulting in feature extraction efficiency and effectiveness that are difficult to achieve. Especially when facing complex and diverse multimodal data, it is difficult to fully utilize the potential value of each modality of data, thus limiting the overall performance improvement of multimodal learning.

[0005] Regarding fusion strategies, existing traditional fusion strategies are mostly fixed patterns, commonly including feature-level fusion (such as concatenation, weighted summation, etc.) and decision-level fusion (such as voting, weighted averaging, etc.). These fixed fusion strategies have also revealed many problems in practical applications. On the one hand, they lack flexibility, failing to adjust the timing and method of fusion according to the dynamic characteristics of the data, the correlation between modalities, and task requirements, thus limiting the fusion effect and failing to fully leverage the synergistic effect of multimodal data. On the other hand, some fusion methods, such as feature-level fusion, may result in excessively high feature dimensions after fusion, thereby increasing the computational complexity of subsequent processing. This is especially prominent when dealing with large-scale multimodal data, where the consumption of computational resources is particularly prominent, which to some extent limits the scalability and efficiency of multimodal learning methods in practical applications. Summary of the Invention

[0006] To address the technical problems of traditional multimodal learning, such as strong dependence on manually designed feature extraction networks, low efficiency of fixed fusion strategies, and high computational cost of traditional neural architecture search, this invention provides a method and apparatus for automatic multimodal modeling and fusion based on neural architecture search. The technical solution is as follows:

[0007] On the one hand, a multimodal automatic modeling and fusion method based on neural architecture search is provided. This method is implemented by a multimodal automatic modeling and fusion device, and includes:

[0008] S1. Obtain multimodal data and task type.

[0009] S2. For each modality in the multimodal data, the corresponding optimal unit architecture is automatically generated based on neural architecture search; wherein, the optimal unit architecture is in the form of a directed acyclic graph.

[0010] S3. Using the optimal unit architecture as a feature extractor, perform feature extraction on each modality data to obtain the high-dimensional features of each modality data, and store the high-dimensional features of each modality data into a shared feature cache pool.

[0011] S4. Analyze the intermodal dependencies at the feature level based on the high-dimensional features of each modal data, generate a dynamically updated correlation heatmap, and obtain a specific fusion strategy based on the correlation heatmap and the fusion strategy decision-maker.

[0012] S5. Construct the optimal fusion network architecture based on the high-dimensional features in the shared feature cache pool and the specific fusion strategy.

[0013] S6. Based on the optimal fusion network architecture and task type, the output layer and loss function are automatically adjusted through the task adaptation module to obtain a network that adapts to the task requirements and completes different downstream tasks.

[0014] Optionally, in S2, for each modality of multimodal data, the corresponding optimal unit architecture is automatically generated based on neural architecture search, including:

[0015] S21. Obtain each modality data in the multimodal data and initialize the candidate operation library; the candidate operation library includes the operation type and the corresponding activation function.

[0016] S22. Organize the components in the candidate operation library into a hypergraph structure to form a searchable network topology.

[0017] S23. The reinforcement learning-based Controller RNN samples subgraph structures from the hypergraph structure and instantiates the operation sequence of the sampled subgraph structures into the unit architecture of each modality.

[0018] S24. Train the subgraph structure, use reinforcement learning search strategy to evaluate the performance of the subgraph structure, and then generate the optimal cell architecture for each modality of data.

[0019] Optionally, in S24, the subgraph structure is trained, and the performance of the subgraph structure is evaluated using a reinforcement learning search strategy to generate the optimal cell architecture for each modality of data, including:

[0020] The weight matrices of all operations are pre-initialized and shared globally, and only the weights of the selected operations are activated for propagation. The sampled subgraph structure is trained using the dataset of the target task, and only the activated parameters are updated. The gradient is calculated using Monte Carlo estimation. The performance of the subgraph structure is evaluated on the validation set, and the performance index is fed back to the controller as a reward signal and normalized by moving average. The parameters of the controller are updated using the proximal policy optimization algorithm, and an entropy regularization term is added to encourage exploration. The parameters of the controller are updated according to the set update frequency until the maximum number of iterations is reached or the reward fluctuation is less than the set threshold, generating the optimal unit architecture for each modality.

[0021] Optionally, in S4, the inter-modal dependencies at the feature level are analyzed based on the high-dimensional features of each modal data, generating a dynamically updated correlation heatmap, including:

[0022] Based on the high-dimensional features of each modality data, determine the dimension and distribution of the modality features, and determine whether the dimension of the modality features is lower than a preset threshold.

[0023] If the threshold is not lower than the preset threshold, a distance metric method is selected to determine the intermodal dependencies and generate a dynamically updated correlation heatmap.

[0024] If the value is below a preset threshold, it is determined whether the distribution of modal features is clear or whether there are significant differences in feature scale. If the distribution of modal features is clear, a statistical correlation method is selected to determine the intermodal dependencies and generate a dynamically updated correlation heatmap. If there are significant differences in feature scale in the distribution of modal features, a distance metric method is selected to determine the intermodal dependencies and generate a dynamically updated correlation heatmap.

[0025] Optionally, in S4, a specific fusion strategy is obtained based on the correlation heatmap and the fusion strategy decision-maker, including:

[0026] Based on the high, medium, and low correlation levels of the modal pairs in the correlation heatmap, different fusion strategies are selected. For highly correlated modal pairs, fusion is performed through shallow feature interaction to ensure that the features of highly correlated modes can jointly influence the model. For medium-correlation modal pairs, hybrid fusion is used to balance contributions. For low-correlation modal pairs, feature fusion is performed at a deep level to reduce interference between low-correlation modal pairs and avoid introducing noise.

[0027] Optionally, S5 constructs an optimal fusion network architecture based on high-dimensional features in a shared feature cache pool and a specific fusion strategy, including:

[0028] S51. Initialize the number of fusion layers Define the maximum number of fusion layers. Initialize a candidate architecture pool to store high-potential architectures selected at each step.

[0029] S52, Regarding the first The layer fusion layer uses feature splicing to combine the first layer... The output of the layer fusion layer and the first Features from the layer fusion layer are fused, and features are allowed to jump between different layers. The fused features are then subjected to a non-linear transformation through an activation function to construct a candidate architecture.

[0030] S53, Regarding the first All candidate architectures under the layer fusion layer are evaluated using the SMBO algorithm based on the sequential model, combined with the LSTM surrogate model to predict the architecture performance.

[0031] S54. Sort all candidate architectures according to performance, select the top-performing architectures as high-potential candidate architectures, and add them to the candidate architecture pool.

[0032] S55. For high-potential candidate architectures, a weight inheritance mechanism is used to initialize parameters from the historical best layer for training.

[0033] S56, at the maximum number of layers Next, all high-potential candidate architectures in the candidate architecture pool are evaluated, and the architecture with the best performance is selected as the optimal fusion network architecture.

[0034] Optionally, in S6, based on the optimal fusion network architecture and task type, the task adaptation module automatically adjusts the output layer and loss function to obtain a network adapted to the task requirements, including:

[0035] S61. Dynamically configure the output layer and loss function for the optimal fusion network architecture according to the task type.

[0036] S62. Perform dynamic network structure adjustment and two-stage training on the configured optimal fusion network architecture; in the first stage, fix the parameters of the task adaptation module and optimize the feature representation of the configured optimal fusion network architecture; in the second stage, lock the parameters of the optimal fusion network architecture and fine-tune the task adaptation module.

[0037] S63. Determine if the downstream task type has changed. If it has changed, unload the output layer and loss function of the current task, load the configuration of the new task, and start hot start fine-tuning. If it has not changed, perform performance monitoring on the optimal fusion network architecture.

[0038] On the other hand, a multimodal automatic modeling and fusion device based on neural architecture search is provided. This device is applied to a multimodal automatic modeling and fusion method based on neural architecture search. The device includes:

[0039] The acquisition module is used to acquire multimodal data and task types.

[0040] The neural architecture search module is used to automatically generate the corresponding optimal unit architecture for each modality in the multimodal data based on neural architecture search; where the optimal unit architecture is in the form of a directed acyclic graph.

[0041] The single-modal feature extraction module is used to extract features from each modal data by using the optimal unit architecture as a feature extractor, thereby obtaining high-dimensional features for each modal data and storing the high-dimensional features of each modal data into a shared feature cache pool.

[0042] The modal correlation calculation module is used to analyze the intermodal dependencies at the feature level based on the high-dimensional features of each modal data, generate a dynamically updated correlation heatmap, and obtain a specific fusion strategy based on the correlation heatmap and the fusion strategy decision-maker.

[0043] The dynamic fusion network module is used to construct the optimal fusion network architecture based on high-dimensional features in the shared feature cache pool and a specific fusion strategy.

[0044] The task adaptation module is used to automatically adjust the output layer and loss function according to the optimal fusion network architecture and task type to obtain a network that adapts to the task requirements and completes different downstream tasks.

[0045] On the other hand, a multimodal automatic modeling and fusion device is provided, the multimodal automatic modeling and fusion device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods in the above-described multimodal automatic modeling and fusion method based on neural architecture search.

[0046] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods of multimodal automatic modeling and fusion based on neural architecture search.

[0047] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following:

[0048] This invention achieves efficient and low-cost end-to-end automatic multimodal modeling by jointly optimizing unimodal feature extraction and multimodal feature fusion through automated architecture search. The core of this method lies in automatically generating lightweight network units to construct an efficient feature extraction network using a neural architecture search mechanism based on reinforcement learning search strategies, while reducing training overhead through parameter sharing. In the multimodal fusion stage, the optimal fusion timing is dynamically determined by quantitatively analyzing the correlation between modalities, overcoming the limitations of traditional fixed strategies. This invention combines progressive search with surrogate model prediction. Progressive search efficiently selects the optimal fusion architecture by gradually increasing the complexity of the search space; surrogate model prediction technology further accelerates the automatic search process by quickly evaluating the performance of candidate architectures through the construction of lightweight surrogate models. Furthermore, this invention can adaptively select the loss function according to the task type, supporting various task requirements. The innovation of this invention lies in proposing a dynamically correlated fusion mechanism to achieve joint search and global optimization of unimodal neural architecture search and multimodal fusion architecture, completing efficient automatic modeling. This method is a general multimodal solution that can be widely applied to various multimodal application scenarios, and has high generalization, strong noise robustness and cross-domain transfer capability. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of a multimodal automatic modeling and fusion method based on neural architecture search provided by an embodiment of the present invention;

[0051] Figure 2 This is a framework diagram of the multimodal automatic modeling and fusion method based on neural architecture joint search provided in the embodiments of the present invention;

[0052] Figure 3 This is a flowchart of the neural architecture search module provided in an embodiment of the present invention;

[0053] Figure 4 This is a flowchart of the multimodal network fusion process provided in an embodiment of the present invention;

[0054] Figure 5 This is a flowchart of the task adaptation module provided in an embodiment of the present invention;

[0055] Figure 6 This is a flowchart of the multimodal automatic modeling and fusion method based on neural architecture search provided in an embodiment of the present invention;

[0056] Figure 7 This is a block diagram of a multimodal automatic modeling and fusion device based on neural architecture search provided in an embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram of the structure of a multimodal automatic modeling and fusion device provided in an embodiment of the present invention. Detailed Implementation

[0058] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0059] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0060] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0061] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0062] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0063] This invention provides a multimodal automatic modeling and fusion method based on neural architecture search. This method can be implemented by a multimodal automatic modeling and fusion device, which can be a terminal or a server. Figure 1 The flowchart shown is for a multimodal automatic modeling and fusion method based on neural architecture search. The processing flow of this method may include the following steps:

[0064] S1. Obtain multimodal data and task type.

[0065] In one feasible implementation, this invention can be applied to the field of computer science, as well as to fields such as the Internet of Things (IoT), Industrial IoT, transportation, and healthcare. It can also be applied to the field of multimodal data fusion, especially in complex scenarios requiring efficient processing and fusion of multiple modalities of data (such as text, images, audio, and video).

[0066] Specifically, in the field of intelligent healthcare, the system automatically constructs a 3D convolutional network through NAS to process CT image sequences, while using the Transformer model to parse electronic medical record text and achieving feature fusion of lesion areas based on dynamically generated correlation heatmaps, providing precise assistance for clinical diagnosis.

[0067] In industrial IoT scenarios, cross-modal correlation analysis is performed on equipment vibration spectrum, infrared thermal imaging, and operation and maintenance logs. Vibration signals are processed by a one-dimensional convolutional network that is automatically optimized by NAS, and thermal imaging features are extracted by combining a visual attention mechanism to build an end-to-end equipment health status prediction model.

[0068] In the field of intelligent transportation, NAS can automatically generate multi-scale feature extraction networks to process vehicle camera data, while searching for the optimal point cloud sparse convolutional architecture to parse LiDAR information, and improve environmental perception capabilities through automatically optimized fusion strategies.

[0069] In smart city management, a spatiotemporal attention mechanism can be introduced to integrate surveillance video streams, social media texts, and traffic flow time-series data to build an urban event early warning system.

[0070] These application solutions demonstrate the unique advantages of this invention in processing diverse and heterogeneous information such as text, images, audio, video, and sensor data. Through automated neural architecture search and intelligent fusion strategies, it provides customized multimodal solutions for various industries.

[0071] like Figure 2 As shown, this invention proposes a multimodal automatic modeling and fusion method based on neural architecture joint search. It achieves efficient modeling from single-modal feature extraction to multimodal fusion through a fully automated process, providing an end-to-end solution for adaptive interaction and efficient joint modeling of cross-modal data.

[0072] S2. For each modality in the multimodal data, the corresponding optimal unit architecture is automatically generated based on neural architecture search; wherein, the optimal unit architecture is in the form of a directed acyclic graph.

[0073] In one feasible implementation, each unimodal unit architecture consists of a dynamic DAG (Directed Acyclic Graph) generated by the controller. The nodes contain candidate operations and activation functions. The controller generates a high-reward structure through a reinforcement learning strategy, and a parameter sharing mechanism is used to reduce computational costs. Monte Carlo gradient estimation is used to update the shared weights, thereby realizing the automatic modeling of the unimodal feature extraction network.

[0074] Optionally, step S2 above may include the following steps S21-S24:

[0075] S21. Obtain each modality data in the multimodal data and initialize the candidate operation library; the candidate operation library includes the operation type and the corresponding activation function.

[0076] S22. Organize the components in the candidate operation library into a hypergraph structure to form a searchable network topology.

[0077] S23. The reinforcement learning-based Controller RNN samples subgraph structures from the hypergraph structure and instantiates the operation sequence of the sampled subgraph structures into the unit architecture of each modality.

[0078] S24. Train the subgraph structure, use reinforcement learning search strategy to evaluate the performance of the subgraph structure, and then generate the optimal cell architecture for each modality of data.

[0079] Specifically, step S24 above may include:

[0080] The weight matrices of all operations are pre-initialized and shared globally, and only the weights of the selected operations are activated for propagation. The sampled subgraph structure is trained using the dataset of the target task, and only the activated parameters are updated. The gradient is calculated using Monte Carlo estimation. The performance of the subgraph structure is evaluated on the validation set, and the performance index is fed back to the controller as a reward signal and normalized by moving average. The parameters of the controller are updated using the proximal policy optimization algorithm, and an entropy regularization term is added to encourage exploration. The parameters of the controller are updated according to the set update frequency until the maximum number of iterations is reached or the reward fluctuation is less than the set threshold, generating the optimal unit architecture for each modality.

[0081] In one feasible implementation, during the neural architecture search process, the search space is first defined, and the types of operations in the candidate operation set are clarified, including different candidate operations, activation functions, and other operations such as identity mapping and zero operations. The unit structure is organized in the form of a directed acyclic graph, where nodes represent operations, edges represent data flow, input and output nodes are defined, and the range of intermediate nodes is set. The controller network adopts a two-layer LSTM (Long Short-Term Memory) structure, with the weight matrix and bias terms randomly initialized to a uniform distribution. The controller's input is the encoded representation of the sampled operation sequence, and its output is the probability distribution of the next operation. During the training and validation phases, a DAG structure is constructed using the operation sequences sampled by the controller to generate specific unit architectures. The weight matrices of all possible operations are pre-initialized and shared globally; only the weights of the selected operations are activated for propagation. The sampled sub-architectures are trained using the target task's dataset, updating only the activated shared parameters, and gradients are calculated using Monte Carlo estimation. The sub-architecture performance is evaluated on the validation set, and the performance metrics are fed back to the controller as a reward signal, followed by moving average normalization. The controller parameters are updated using the Proximal Policy Optimization (PPO) algorithm, with an entropy regularization term added to encourage exploration. The controller parameters are updated according to a set update frequency. Termination conditions include reaching the maximum number of iterations or the reward fluctuation being less than a set threshold. During the search, the performance metrics of each sampled sub-architecture are recorded, and the best-performing unit architecture is selected as the final architecture at the end of the search. If the performance of a sub-architecture is significantly better than other sub-architectures and the performance improvement is very small in consecutive iterations, the search can be terminated early. Next, the modality correlation calculation module dynamically decides the timing and method of fusion to achieve effective fusion of multimodal features.

[0082] Neural architecture search process, such as Figure 3 As shown, the specific steps are as follows:

[0083] ① Input the raw data of each modality into the system and initialize the candidate operation library, including possible operation types and their corresponding activation functions;

[0084] ② Organize the components in the operation library into a hypergraph structure to form a searchable network topology.

[0085] ③ The reinforcement learning-based Controller RNN samples subgraph structures from the hypergraph and instantiates the sampled operation sequence into specific network structures (modal unit architectures).

[0086] ④ Train subgraphs and evaluate their performance, provide feedback optimization signals, use reinforcement learning search strategies, use the structure of the unit as the action space, and evaluate the performance of different structures through a reward mechanism.

[0087] ⑤ Record the accuracy of the validation set and use it as a reward signal to feed back to the Controller RNN to evaluate the performance of different unit architectures and select the architecture with the best performance.

[0088] ⑥ After finding the optimal cell structure, use the optimized cell architecture for feature extraction.

[0089] This invention utilizes neural architecture search technology to automatically identify the optimal unimodal network unit. This unimodal network unit is then used as a feature extractor for multimodal feature fusion. The method automatically selects and adjusts the multimodal feature fusion architecture, leveraging the complementarity and correlation between modalities to maximize information integration and improve the performance and efficiency of multimodal data processing. Through intelligent architecture search, the method can adaptively fuse data for different data types and tasks, avoiding the complexity of manual design and effectively improving the accuracy and robustness of multimodal learning systems in practical applications.

[0090] Furthermore, this invention proposes a neural architecture search mechanism based on directed acyclic graphs, constructing a search space with shareable sub-modules containing multiple ordered connected neural network operation nodes. Each node selects the optimal operation from a preset set of operation candidates and defines the feature flow direction through directed edges. A controller network is used to sample the optimal architecture in the search space through a reinforcement learning strategy. Through a parameter sharing mechanism, all sampled sub-architectures share weight parameters. The controller network and sub-architecture parameters are trained alternately to optimize the overall model and automatically generate the optimal lightweight network unit.

[0091] S3. Using the optimal unit architecture as a feature extractor, perform feature extraction on each modality data to obtain the high-dimensional features of each modality data, and store the high-dimensional features of each modality data into a shared feature cache pool.

[0092] In one feasible implementation, during the feature extraction and fusion configuration stage, the optimal cell architecture found through searching is used as a feature extractor to extract features from the input data, extracting features from different layers as needed. The extracted features can be fixed-length vector representations, and then the extracted features are normalized to ensure the comparability of features from different modalities.

[0093] This invention uses automatically generated lightweight network units as the core feature extractor, which can adapt to the feature extraction requirements of different modal data, maximize feature expression ability while maintaining computational efficiency. Multiple units are connected in cascade or parallel to form a complete feature extraction network, realizing multi-level feature abstraction and representation learning of input data.

[0094] S4. Analyze the intermodal dependencies at the feature level based on the high-dimensional features of each modal data, generate a dynamically updated correlation heatmap, and obtain a specific fusion strategy based on the correlation heatmap and the fusion strategy decision-maker.

[0095] Optionally, in S4, the inter-modal dependencies at the feature level are analyzed based on the high-dimensional features of each modal data, generating a dynamically updated correlation heatmap, including:

[0096] Based on the high-dimensional features of each modality data, determine the dimension and distribution of the modality features, and determine whether the dimension of the modality features is lower than a preset threshold.

[0097] If the threshold is not lower than the preset threshold, a distance metric method is selected to determine the intermodal dependencies and generate a dynamically updated correlation heatmap.

[0098] If the value is below a preset threshold, it is determined whether the distribution of modal features is clear or whether there are significant differences in feature scale. If the distribution of modal features is clear, a statistical correlation method is selected to determine the intermodal dependencies and generate a dynamically updated correlation heatmap. If there are significant differences in feature scale in the distribution of modal features, a distance metric method is selected to determine the intermodal dependencies and generate a dynamically updated correlation heatmap.

[0099] Optionally, in S4, a specific fusion strategy is obtained based on the correlation heatmap and the fusion strategy decision-maker, including:

[0100] Based on the high, medium, and low correlation levels of the modal pairs in the correlation heatmap, different fusion strategies are selected. For highly correlated modal pairs, fusion is performed through shallow feature interaction to ensure that the features of highly correlated modes can jointly influence the model. For medium-correlation modal pairs, hybrid fusion is used to balance contributions. For low-correlation modal pairs, feature fusion is performed at a deep level to reduce interference between low-correlation modal pairs and avoid introducing noise.

[0101] In one feasible implementation, the modality correlation calculation module plays a crucial role in the multimodal fusion system. Its main processes include feature extraction, caching, correlation calculation, and fusion strategy adjustment. The system automatically extracts features from each modality using lightweight units and stores these features in a feature cache pool. Each modality's features are allocated to a corresponding cache space for later use. Then, the system activates the modality correlation calculation module to calculate the dependencies between modalities. First, the dimensionality and distribution of the modality features are determined. If the modality features have low dimensionality and a clear distribution, a statistical correlation method is selected, using mutual information or covariance matrices to quantify the dependencies between different modalities, intuitively reflecting the overall degree of correlation between modalities. If the modality features have high dimensionality or significant differences in feature scale, a distance metric method is selected, measuring the similarity or difference between feature vectors by calculating Euclidean distance, cosine similarity, and Manhattan distance. The correlation results are displayed in the form of a heatmap, and different fusion strategies are selected based on the high, medium, and low correlation levels in the heatmap. For highly correlated modal pairs, early fusion is employed to capture strong dependencies. Fusion is performed through shallow feature interactions (such as feature concatenation or weighted summation) to ensure that features from highly correlated modalities collectively influence the model. For moderately correlated modalities, a hybrid fusion approach is used to balance contributions, including multi-layer feature interactions such as mid-level feature concatenation and weighted attention mechanisms. For low-correlation modalities, late fusion is chosen, i.e., feature fusion is performed at a deep level to reduce interference between low-correlation modalities and avoid introducing noise. Furthermore, the system periodically recalculates the correlation between modalities based on changes in feature representations, adjusts the fusion strategy in real time, and dynamically updates the heatmap to ensure optimal performance. If it is uncertain which method is more suitable, both metrics can be calculated simultaneously, and experimental results can be combined for comprehensive analysis to ensure that the selected fusion strategy is optimal.

[0102] This invention evaluates fusion benefits in real time based on intermodal mutual information calculation. It intelligently selects the optimal fusion strategy by dynamically calculating the correlation between modalities and analyzing task requirements. For highly correlated modalities, an early fusion strategy is employed to fully exploit complementary information; for moderately correlated modalities, mid-level abstract features are selectively fused to balance modality-specificity preservation with cross-modal correlation modeling; and for low-correlation modalities, a late fusion strategy is used to reduce information redundancy. By evaluating intermodal dependencies in real time and dynamically adjusting the fusion method, optimal information integration efficiency is ensured across different tasks and datasets, significantly reducing computational overhead while improving model accuracy.

[0103] The dynamic correlation-guided fusion timing decision method achieves adaptive fusion by quantifying the interaction strength between modalities. This method calculates the correlation of modal features based on normalized mutual information and dynamically selects early or late fusion strategies. Highly correlated modalities preferentially connect shallow features to capture low-level associations, while lowly correlated modalities avoid noise interference through deep feature interactions, thereby significantly improving cross-modal semantic alignment capability and noise robustness.

[0104] S5. Construct the optimal fusion network architecture based on the high-dimensional features in the shared feature cache pool and the specific fusion strategy.

[0105] Optionally, step S5 above may include the following steps S51-S56:

[0106] S51. Initialize the number of fusion layers Define the maximum number of fusion layers. Initialize a candidate architecture pool to store high-potential architectures selected at each step.

[0107] S52, Regarding the first The layer fusion layer uses feature splicing to combine the first layer... The output of the layer fusion layer and the first Features from the layer fusion layer are fused, and features are allowed to jump between different layers. The fused features are then subjected to a non-linear transformation through an activation function to construct a candidate architecture.

[0108] S53, Regarding the first All candidate architectures under the layer fusion layer are evaluated using the SMBO algorithm based on the sequential model, combined with the LSTM surrogate model to predict the architecture performance.

[0109] S54. Sort all candidate architectures according to performance, select the top-performing architectures as high-potential candidate architectures, and add them to the candidate architecture pool.

[0110] S55. For high-potential candidate architectures, a weight inheritance mechanism is used to initialize parameters from the historical best layer for training.

[0111] S56, at the maximum number of layers Next, all high-potential candidate architectures in the candidate architecture pool are evaluated, and the architecture with the best performance is selected as the optimal fusion network architecture.

[0112] In one feasible implementation, after calculating the correlation between modalities, the fusion operation determines how to merge features based on the correlation. Different fusion strategies utilize modal correlations for weighted fusion or adjust the way information flows. At each fusion layer, features from different modalities are merged using feature concatenation, which can enhance cross-modal information exchange through weighted concatenation, addition, etc. Each fusion layer not only fuses the features of the current layer with the output of the previous layer, but also performs nonlinear transformations through activation functions to obtain a comprehensive multimodal representation. The fusion operation at each layer interacts the output of the previous layer with the multimodal features of the current layer, thereby achieving more flexible message passing. Skip connections can directly pass important features, avoiding feature loss or degradation. After the output of each fusion layer, a nonlinear transformation is introduced using an activation function to enhance the network's expressive power. The multimodal feature fusion configuration provides a flexible way to process data from different modalities. By defining the strategy for selecting specific features from each modality for fusion through the configuration matrix, as well as the nonlinear activation function used in each fusion layer, the feature processing method can be dynamically adjusted according to task requirements.

[0113] A unimodal neural architecture search unit is used as a pre-trained multi-layer feature extractor, assuming that... There are 3 modes, each with a function. , It is the input data. It is a neural network composed of multiple sub-functions. Introducing a first... A neural network is used to fuse multimodal data. Each fusion layer... Output Defined by the following formula:

[0114] (1)

[0115] in, Indicates the first The output of the fusion layer It is the first The search targets of each fusion layer represent a certain intermediate hidden feature in each modality and the activation function used. The intermediate layer features of each modality are extracted through the optimal unit architecture generated by the neural architecture search module. It is the output of the previous fusion layer; for the first fusion layer... Then there is none. . Represents the candidate fusion function. This indicates that the fusion operation selects an index. It is the applied non-linear activation function, determined by the variable Sure.

[0116] Furthermore, the dynamic fusion network module constructs a cross-modal interaction architecture through a progressive search strategy, achieving efficient and flexible multimodal feature fusion through a phased training process. The entire training process includes two key stages: search and final training. In the search stage, the algorithm starts with the simplest fusion architecture, i.e., one fusion layer (…). Starting with [previous iteration], the number of fusion layers is gradually increased to explore more complex architectures. During each iteration, the system samples and trains based on the surrogate model. The system evaluates candidate architectures, and the proxy model predicts the performance of new architectures by learning from historical architecture performance data. To balance exploration and leveraging existing knowledge, a temperature parameter is introduced. The system employs a temperature adjustment mechanism: an initial higher temperature value makes sampling more random, while gradually lowering the temperature as the search progresses, making sampling more reliant on the prediction results of the surrogate model. Simultaneously, the system uses a weight-sharing mechanism to reduce computational costs, allowing sampling architectures with similar structures to share the same layer's weight matrix. Each iteration records the performance of the sampling architecture, continuously optimizing the predictive capabilities of the surrogate model. After several iterations, the search phase ends, and the system selects the best-performing model. The candidate architectures enter the final training phase. In this phase, they undergo more thorough training, including increasing training cycles and expanding the weight matrix size, to ensure their performance is fully realized. Unlike the search phase, the final training phase not only trains the weights of the fusion layer but also fine-tunes the weights of the feature extractor. By using a multi-task loss function, the output of each modality and the fused output are optimized simultaneously, ensuring optimal performance for the entire network. Finally, the performance of all candidate architectures is evaluated on the validation set, and the best-performing architecture is selected as the final multimodal fusion model output for subsequent tasks. This training method, by combining an intelligent search strategy and thorough training, can find the optimal architecture in a vast search space while ensuring the model's performance in real-world tasks.

[0117] The workflow of a multimodal fusion network is as follows: Figure 4 As shown:

[0118] Detailed description of each step in the flowchart:

[0119] ① Initialize the number of fusion layers. Set initial parameters to increase the number of fusion layers. Initialize to 1, indicating that fusion will begin with a single layer. Also, define the maximum number of layers. This indicates a gradual expansion to a maximum. Layer fusion. Initialize the candidate architecture pool by creating an empty pool to store high-potential architectures selected at each step.

[0120] ② Construct a cross-modal interaction architecture. At the current fusion layer... Next, a fusion layer is constructed to support cross-modal skip connections and reuse of features from historical layers. Each fusion layer can receive features from different modalities and allows features to skip connections between different layers while reusing features from historical layers.

[0121] ③ Performance Prediction. Using the SMBO algorithm: The Sequential Model Basic Optimization (SMBO) algorithm is employed, combined with an LSTM surrogate model to predict the performance of untrained architectures. The SMBO algorithm evaluates the performance of candidate architectures sequentially and utilizes a surrogate model (such as LSTM) to predict the performance of untrained architectures. The LSTM surrogate model learns patterns in architecture performance based on historical data; its input is the feature representation of the architecture, and its output is the predicted performance metrics such as accuracy and loss. For the current number of fusion layers... For all candidate architectures, use the LSTM proxy model to predict their performance.

[0122] ④ Screening high-potential candidate architectures: Based on the performance predicted by the LSTM surrogate model, all candidate architectures are ranked, and the top-3 architectures with the highest performance are selected as high-potential candidate architectures. These architectures are considered to have good performance potential in subsequent training. The selected high-potential candidate architectures are added to the candidate architecture pool for subsequent training and evaluation.

[0123] ⑤ For the selected high-potential candidate architectures, a weight inheritance mechanism is adopted to initialize parameters from the historical best layer. The parameters of the previously trained best layer are used as the initial parameters of the current architecture, reducing redundant computation and improving training efficiency. When initializing parameters, the parameters of the historical best layer are reused to avoid starting from random initialization, thereby accelerating the training process.

[0124] ⑥ Output the optimal fusion architecture and evaluate the final candidate architecture. (At the maximum number of layers) Next, a final evaluation is performed on all architectures in the candidate architecture pool, and the architecture with the best performance is selected for subsequent multimodal data processing tasks.

[0125] In this invention, a progressive search strategy optimizes search efficiency by gradually increasing the complexity of the fusion network. Starting from a single-layer fusion architecture, the number of layers is gradually increased while inheriting historically optimal parameter weights. Combined with an early stopping mechanism, redundant computation is avoided, thereby improving the efficiency of automatic modeling and fusion architecture search.

[0126] End-to-end training employs a two-stage optimization strategy to achieve global parameter coordination. The first stage freezes the single-modal network to quickly converge the fusion layer, and the second stage jointly fine-tunes all parameters and introduces a dynamic learning rate and weight decay mechanism to support flexible adaptation to various tasks.

[0127] This invention constructs a dynamically configurable multimodal feature fusion network architecture. Based on the correlation calculation results between modalities, it automatically designs flexible feature fusion strategies and creates a candidate architecture pool covering various possible fusion configurations, such as feature concatenation and weighted summation. The most suitable fusion configuration for the current data and task is dynamically selected using correlation analysis results. Then, a feature fusion module is designed, capable of performing corresponding operations according to different fusion strategies and easily extensible to adapt to new fusion strategies. Cross-layer skip connections in the network enhance feature interaction, and nonlinear activation functions are integrated to increase the network's expressive power. A complete multimodal fusion network architecture including an automatic feature extraction layer, intelligent decision-making, and fusion layers is constructed. This network is trained using an end-to-end training and multi-task learning strategy, and its performance is evaluated on a validation set. The fusion strategy and network structure are adjusted based on feedback, forming a dynamic, configurable, and adaptive multimodal automatic modeling and fusion system that adapts to different modal data and task requirements.

[0128] S6. Based on the optimal fusion network architecture and task type, the output layer and loss function are automatically adjusted through the task adaptation module to obtain a network that adapts to the task requirements and completes different downstream tasks.

[0129] Optionally, step S6 above may include the following steps S61-S63:

[0130] S61. Dynamically configure the output layer and loss function for the optimal fusion network architecture according to the task type.

[0131] S62. Perform dynamic network structure adjustment and two-stage training on the configured optimal fusion network architecture; in the first stage, fix the parameters of the task adaptation module and optimize the feature representation of the configured optimal fusion network architecture; in the second stage, lock the parameters of the optimal fusion network architecture and fine-tune the task adaptation module.

[0132] S63. Determine if the downstream task type has changed. If it has changed, unload the output layer and loss function of the current task, load the configuration of the new task, and start hot start fine-tuning. If it has not changed, perform performance monitoring on the optimal fusion network architecture.

[0133] In one feasible implementation, the task adaptation module is responsible for automatically adjusting the output layer and loss function based on the optimal fusion architecture output by the multimodal fusion network to adapt to the needs of different downstream tasks. This module can select the most suitable configuration according to different task types and perform two-stage training optimization. The workflow of the task adaptation module is as follows: Figure 5 As shown:

[0134] ① Load the best converged architecture obtained during the architecture search phase.

[0135] ② Dynamically configure the output layer and loss function according to the task type. For example, for classification tasks, use a Softmax or Sigmoid output layer with label smoothing cross-entropy loss; for regression tasks, use a linear output layer and adaptive Huber loss; and for generation tasks, configure a Transformer decoder and combine adversarial loss with L1 reconstruction loss.

[0136] ③ Perform dynamic network structure adjustment and two-stage training. The first stage fixes the parameters of the task adaptation module and focuses on optimizing the feature representation of the fusion network. The second stage locks the parameters of the fusion network and focuses on fine-tuning the task adaptation module. This two-stage separate training method ensures the independence of feature learning while improving training efficiency.

[0137] ④ Real-time task switching and performance monitoring are implemented. When a change in the downstream task type is detected, the module immediately unloads the output layer and loss function of the current task, loads the configuration of the new task, and initiates a hot-start fine-tuning process. This process is based on historical fusion parameters and does not require retraining. At the same time, the system monitors indicators such as loss value and accuracy in real time to ensure system stability. When performance fluctuations exceed the threshold, an alarm and version rollback mechanism will be automatically triggered.

[0138] like Figure 6 As shown, this invention achieves efficient automatic single-modal feature extraction and multimodal fusion strategies through automated architecture search and joint optimization. The system first automatically generates optimal searchable lightweight single-modal unit structures for different modal data, enabling efficient automatic single-modal modeling. After finding the optimal unit structure, feature extraction is performed using the optimized unit architecture, and the extracted single-modal features are fused and configured. The fusion configuration, combined with the output of the previous fusion layer and a nonlinear activation function, constitutes the multimodal fusion search space. A progressive search strategy, combined with the SMBO algorithm, gradually expands the complexity from simple architectures. An LSTM surrogate model is used to predict the performance of untrained architectures to reduce search time. Finally, the loss function and output layer are adaptively selected based on the task type, achieving efficient end-to-end multimodal fusion network model training and multimodal feature fusion.

[0139] In this invention, efficient and low-cost end-to-end automatic multimodal modeling is achieved by jointly optimizing unimodal feature extraction and multimodal feature fusion through automated architecture search. The core of this method lies in automatically generating lightweight network units to construct an efficient feature extraction network using a neural architecture search mechanism based on reinforcement learning search strategies, while reducing training overhead through a parameter sharing mechanism. In the multimodal fusion stage, the optimal fusion timing is dynamically determined by quantitatively analyzing the correlation between modalities, overcoming the limitations of traditional fixed strategies. This invention combines progressive search with surrogate model prediction. Progressive search efficiently selects the optimal fusion architecture by gradually increasing the complexity of the search space; surrogate model prediction technology quickly evaluates the performance of candidate architectures by constructing lightweight surrogate models, further accelerating the automatic search process. Furthermore, this invention can adaptively select the loss function according to the task type, supporting various task requirements. The innovation of this invention lies in proposing a dynamically correlated fusion mechanism to achieve joint search and global optimization of unimodal neural architecture search and multimodal fusion architecture, completing efficient automatic modeling. This method is a general multimodal solution that can be widely applied to various multimodal application scenarios, and has high generalization, strong noise robustness and cross-domain transfer capability.

[0140] Figure 7 This is a block diagram illustrating a multimodal automatic modeling and fusion apparatus based on neural architecture search, according to an exemplary embodiment. The apparatus is used for a multimodal automatic modeling and fusion method based on neural architecture search. (Refer to...) Figure 7 The device includes an acquisition module 310, a neural architecture search module 320, a single-modal feature extraction module 330, a modality correlation calculation module 340, a dynamic fusion network module 350, and a task adaptation module 360. Among them:

[0141] The acquisition module 310 is used to acquire multimodal data and task types.

[0142] The neural architecture search module 320 is used to automatically generate the corresponding optimal unit architecture for each modality in the multimodal data based on neural architecture search; wherein the optimal unit architecture is in the form of a directed acyclic graph.

[0143] The single-modal feature extraction module 330 is used to use the optimal unit architecture as a feature extractor to extract features from each modal data, obtain high-dimensional features of each modal data, and store the high-dimensional features of each modal data into a shared feature cache pool.

[0144] The modal correlation calculation module 340 is used to analyze the intermodal dependencies at the feature level based on the high-dimensional features of each modal data, generate a dynamically updated correlation heatmap, and obtain a specific fusion strategy based on the correlation heatmap and the fusion strategy decision-maker.

[0145] The dynamic fusion network module 350 is used to construct the optimal fusion network architecture based on high-dimensional features in the shared feature cache pool and a specific fusion strategy.

[0146] The task adaptation module 360 ​​is used to automatically adjust the output layer and loss function according to the optimal fusion network architecture and task type to obtain a network that adapts to the task requirements and completes different downstream tasks.

[0147] In one feasible implementation, the neural architecture search module automatically generates lightweight unit structures based on neural architecture search. Each unit contains searchable neural network operations. An optimal operation sequence is dynamically selected from the candidate operation set using a reinforcement learning controller, and the candidate architecture is trained using a weight-sharing mechanism. Differentiated search constraints are designed for different modal characteristics. The output optimal unit architecture is stored in the form of a directed acyclic graph, supporting flexible stacking and expansion later.

[0148] Single-modal feature extraction module: The optimal unit obtained through the search is used as the feature extractor. The optimized unit has better feature extraction capabilities and model performance. It can be flexibly stacked to build larger or smaller networks, further enhancing the feature extraction capabilities and learning more complex feature representations. Each modality's data is processed by the corresponding unit network, and the extracted high-dimensional features are stored in a shared feature cache pool to provide standardized input for subsequent fusion.

[0149] Modality correlation calculation module: After feature extraction of each modality's data is completed through automatically generated lightweight units and stored in the feature cache pool, the system starts the modality correlation calculation module. Based on statistical correlation methods or distance metric methods, it quantifies and analyzes the inter-modal dependencies at the feature level, generating a dynamically updated correlation heatmap. The calculation results directly drive the fusion strategy decision-maker to select the optimal fusion method. By periodically recalculating to adapt to the dynamic evolution of feature representation capabilities, it ensures that modality correlation calculation is performed at the stage of optimal feature quality, while decoupling from the architecture search process to reduce complexity.

[0150] The dynamic fusion network module receives multimodal features generated by the single-modal feature extraction module from the feature cache pool. It then initializes these features using a specific fusion strategy provided by the modality correlation calculation module, constructing a minimum feasible architecture containing basic units and a simple fusion layer. During progressive scaling, the system dynamically generates a candidate architecture pool through a feature fusion configuration step, inserts higher-order components such as attention mechanisms, and evaluates them using Sequence Model Optimization (SMBO). Only the top 20% of potential architectures are actually trained. The search stops when the performance improvement is less than 1% after three consecutive iterations or when a preset complexity threshold is reached. Finally, the optimal fusion network architecture, compatible with the task adaptation module, is output. This module achieves efficient feature interaction through a weight sharing mechanism and dynamic strategy-driven approach, ensuring end-to-end collaborative optimization from feature extraction to result output.

[0151] Task Adaptation Module: This module is responsible for automatically adjusting the output layer and loss function based on the optimal fusion architecture output by the multimodal fusion network to adapt to the needs of different downstream tasks. It selects the most suitable configuration according to the different types of tasks and performs a two-stage training optimization. The first stage freezes the parameters of the single-modal network to quickly converge the fusion layer; the second stage jointly fine-tunes all parameters and improves generalization ability through dynamic learning rate and weight decay. An end-to-end joint optimization mechanism feeds the fusion network performance gradient back to the single-modal unit generation process through a NAS controller, achieving collaborative optimization of feature extraction and fusion strategies.

[0152] In this invention, efficient and low-cost end-to-end automatic multimodal modeling is achieved by jointly optimizing unimodal feature extraction and multimodal feature fusion through automated architecture search. The core of this method lies in automatically generating lightweight network units to construct an efficient feature extraction network using a neural architecture search mechanism based on reinforcement learning search strategies, while reducing training overhead through a parameter sharing mechanism. In the multimodal fusion stage, the optimal fusion timing is dynamically determined by quantitatively analyzing the correlation between modalities, overcoming the limitations of traditional fixed strategies. This invention combines progressive search with surrogate model prediction. Progressive search efficiently selects the optimal fusion architecture by gradually increasing the complexity of the search space; surrogate model prediction technology quickly evaluates the performance of candidate architectures by constructing lightweight surrogate models, further accelerating the automatic search process. Furthermore, this invention can adaptively select the loss function according to the task type, supporting various task requirements. The innovation of this invention lies in proposing a dynamically correlated fusion mechanism to achieve joint search and global optimization of unimodal neural architecture search and multimodal fusion architecture, completing efficient automatic modeling. This method is a general multimodal solution that can be widely applied to various multimodal application scenarios, and has high generalization, strong noise robustness and cross-domain transfer capability.

[0153] Figure 8This is a schematic diagram of the structure of a multimodal automatic modeling and fusion device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the multimodal automatic modeling and fusion equipment may include the above-mentioned Figure 7 The illustrated multimodal automatic modeling and fusion device is based on neural architecture search. Optionally, the multimodal automatic modeling and fusion device 410 may include a first processor 2001.

[0154] Optionally, the multimodal automatic modeling and fusion device 410 may also include a memory 2002 and a transceiver 2003.

[0155] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0156] The following is combined with Figure 8 A detailed introduction to each component of the multimodal automatic modeling and fusion device 410 is provided below:

[0157] The first processor 2001 is the control center of the multimodal automatic modeling and fusion device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0158] Optionally, the first processor 2001 can perform various functions of the multimodal automatic modeling and fusion device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0159] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 8 CPU0 and CPU1 are shown in the diagram.

[0160] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0161] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal automatic modeling and fusion method based on neural architecture search, characterized in that, The method includes: S1. Obtain multimodal data and task type; where multimodal data includes text, images, audio and / or video; S2. For each modality of multimodal data, the corresponding optimal unit architecture is automatically generated based on neural architecture search; wherein, the optimal unit architecture is in the form of a directed acyclic graph. S3. Using the optimal unit architecture as a feature extractor, perform feature extraction on each modal data to obtain high-dimensional features of each modal data, and store the high-dimensional features of each modal data into a shared feature cache pool. S4. Analyze the intermodal dependencies at the feature level based on the high-dimensional features of each modal data, generate a dynamically updated correlation heatmap, and obtain a specific fusion strategy based on the correlation heatmap and the fusion strategy decision-maker. S5. Construct the optimal fusion network architecture based on the high-dimensional features in the shared feature cache pool and the specific fusion strategy; S6. Based on the optimal fusion network architecture and task type, the output layer and loss function are automatically adjusted through the task adaptation module to obtain a network that adapts to the task requirements and completes different downstream tasks. The step S5, which involves constructing the optimal fusion network architecture based on high-dimensional features in the shared feature cache pool and a specific fusion strategy, includes: S51. Initialize the number of fusion layers, define the maximum number of fusion layers, and initialize the candidate architecture pool to store the high-potential architectures selected in each step. S52. For the i-th fusion layer, the output of the i-th fusion layer is fused with the features of the i-th fusion layer by feature concatenation, and the features are allowed to jump between different levels. The fused features are then transformed non-linearly through an activation function to construct a candidate architecture. S53. For all candidate architectures under the fusion layer, the SMBO algorithm based on the sequential model is used, combined with the LSTM proxy model to predict the performance of the architecture. S54. Sort all candidate architectures according to performance, select the architectures with the highest performance as high-potential candidate architectures, and add them to the candidate architecture pool. S55. For high-potential candidate architectures, a weight inheritance mechanism is used to initialize parameters from the historical best layer for training. S56. At the maximum number of layers, evaluate all high-potential candidate architectures in the candidate architecture pool and select the architecture with the best performance as the optimal fusion network architecture. The step S6, which involves automatically adjusting the output layer and loss function based on the optimal fusion network architecture and task type using a task adaptation module to obtain a network adapted to the task requirements, includes: S61. Dynamically configure the output layer and loss function for the optimal fusion network architecture according to the task type; S62. Perform dynamic network structure adjustment and two-stage training on the configured optimal fusion network architecture; in the first stage, fix the parameters of the task adaptation module and optimize the feature representation of the configured optimal fusion network architecture; in the second stage, lock the parameters of the optimal fusion network architecture and fine-tune the task adaptation module. S63. Determine if the downstream task type has changed. If it has changed, unload the output layer and loss function of the current task, load the configuration of the new task, and start hot start fine-tuning. If it has not changed, perform performance monitoring on the optimal fusion network architecture.

2. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 1, characterized in that, In S2, for each modality of multimodal data, the corresponding optimal unit architecture is automatically generated based on neural architecture search, including: S21. Obtain each modality data in the multimodal data and initialize the candidate operation library; the candidate operation library includes the operation type and the corresponding activation function; S22. Organize the components in the candidate operation library into a hypergraph structure to form a searchable network topology; S23. The reinforcement learning-based Controller RNN samples subgraph structures from the hypergraph structure and instantiates the operation sequence of the sampled subgraph structures into the unit architecture of each modality. S24. Train the subgraph structure, use reinforcement learning search strategy to evaluate the performance of the subgraph structure, and then generate the optimal cell architecture for each modality of data.

3. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 2, characterized in that, The process in S24 involves training the subgraph structure, evaluating its performance using a reinforcement learning search strategy, and generating the optimal unit architecture for each modality of data, including: The weight matrices of all operations are pre-initialized and shared globally, and only the weights of the selected operations are activated for propagation. The sampled subgraph structure is trained using the dataset of the target task, and only the activated parameters are updated. The gradient is calculated using Monte Carlo estimation. The performance of the subgraph structure is evaluated on the validation set, and the performance index is fed back to the controller as a reward signal and normalized by moving average. The parameters of the controller are updated using the proximal policy optimization algorithm, and an entropy regularization term is added to encourage exploration. The parameters of the controller are updated according to the set update frequency until the maximum number of iterations is reached or the reward fluctuation is less than the set threshold, generating the optimal unit architecture for each modality.

4. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 1, characterized in that, The step S4, which analyzes the inter-modal dependencies at the feature level based on the high-dimensional features of each modal data and generates a dynamically updated correlation heatmap, includes: The dimension and distribution of the modal features are determined based on the high-dimensional features of each modal data, and it is determined whether the dimension of the modal features is lower than a preset threshold. If the value is not lower than the preset threshold, a distance metric method is selected to determine the intermodal dependencies and generate a dynamically updated correlation heatmap. If the value is below a preset threshold, it is determined whether the distribution of modal features is clear or whether there are significant differences in feature scale. If the distribution of modal features is clear, a statistical correlation method is selected to determine the intermodal dependencies and generate a dynamically updated correlation heatmap. If there are significant differences in feature scale in the distribution of modal features, a distance metric method is selected to determine the intermodal dependencies and generate a dynamically updated correlation heatmap.

5. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 1, characterized in that, The specific fusion strategy obtained in S4 based on the correlation heatmap and the fusion strategy decision-maker includes: Based on the high, medium, and low correlation levels of the modal pairs in the correlation heatmap, different fusion strategies are selected. For highly correlated modal pairs, fusion is performed through shallow feature interaction to ensure that the features of highly correlated modes can jointly influence the model. For medium-correlation modal pairs, hybrid fusion is used to balance contributions. For low-correlation modal pairs, feature fusion is performed at a deep level to reduce interference between low-correlation modal pairs and avoid introducing noise.

6. A multimodal automatic modeling and fusion device based on neural architecture search, wherein the multimodal automatic modeling and fusion device based on neural architecture search is used to implement the multimodal automatic modeling and fusion method based on neural architecture search as described in any one of claims 1-5, characterized in that, The device includes: The acquisition module is used to acquire multimodal data and task types; The neural architecture search module is used to automatically generate the corresponding optimal unit architecture for each modality in the multimodal data based on neural architecture search; wherein, the optimal unit architecture is in the form of a directed acyclic graph; The single-modal feature extraction module is used to use the optimal unit architecture as a feature extractor to extract features from each modal data, obtain high-dimensional features of each modal data, and store the high-dimensional features of each modal data into a shared feature cache pool. The modal correlation calculation module is used to analyze the intermodal dependencies at the feature level based on the high-dimensional features of each modal data, generate a dynamically updated correlation heatmap, and obtain a specific fusion strategy based on the correlation heatmap and the fusion strategy decision-maker. The dynamic fusion network module is used to construct the optimal fusion network architecture based on high-dimensional features in the shared feature cache pool and a specific fusion strategy. The task adaptation module is used to automatically adjust the output layer and loss function according to the optimal fusion network architecture and task type to obtain a network that adapts to the task requirements and completes different downstream tasks.

7. A multimodal automatic modeling and fusion device, characterized in that, The multimodal automatic modeling and fusion equipment includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 5.

Images