Neural architecture search-based multi-modal automatic modeling and fusion method and device

Abstract

The invention provides a neural architecture search-based multi-modal automatic modeling and fusion method and device, and relates to the technical field of computer science. The method comprises the following steps: acquiring multi-modal data and a task type; automatically generating a corresponding optimal unit architecture for each modal data based on neural architecture search, and performing feature extraction on each modal data; according to a high-dimensional feature of each modal data, analyzing a dependency relationship between modals of a feature level to generate a dynamically updated correlation thermodynamic diagram, obtaining a specific fusion strategy according to a fusion strategy decision maker, and constructing an optimal fusion network architecture; and an output layer and a loss function are automatically adjusted according to task types, a network meeting task requirements is obtained, and different downstream tasks are completed. According to the method, end-to-end joint optimization of single-mode feature extraction and a multi-mode fusion strategy can be realized, the manual intervention cost and GPU resource consumption are remarkably reduced, and meanwhile, the task performance and the model generalization ability in a complex scene are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer science and technology, and in particular to a multimodal automatic modeling and fusion method and device based on neural architecture search. Background Art

[0002] Most existing multimodal learning methods use manually designed feature extraction networks and fixed fusion strategies. These traditional methods have many bottlenecks that are difficult to break through in practical applications.

[0003] In the field of neural architecture search, NAS (Neural Architecture Search), a technique for automatically optimizing network architectures, has achieved some success in single-modal learning. However, existing neural architecture search methods still face numerous challenges and issues when applied to multimodal feature fusion. First, the computational cost is extremely high. Traditional neural architecture search methods typically require extensive computing resources to conduct large-scale searches to discover the optimal network architecture. Some NAS methods can require thousands of GPU hours to complete the search process, which is prohibitive for most researchers and practical applications, significantly limiting their widespread application in multimodal learning. Second, existing NAS methods primarily focus on optimizing single-modal feature extraction networks or optimize multimodal fusion strategies as independent modules, making it difficult to jointly optimize single-modal feature extraction and multimodal fusion strategies. This separate optimization approach fails to comprehensively consider the characteristics of multimodal data and the interrelationship between feature extraction and fusion. Consequently, it is difficult to achieve optimal multimodal modeling results and fails to fully leverage the overall advantages of multimodal data.

[0004] In terms of feature extraction, existing traditional multimodal learning methods often rely on manually designed network architectures. For example, for image data, a predefined CNN (Convolutional Neural Network) architecture is often used to extract features; for text data, RNN (Recurrent Neural Network) and its variants (such as LSTM and GRU) are often used to process sequential information. However, these manually designed feature extraction networks are highly dependent on the experience and knowledge of domain experts. New data types or specific tasks require redesigning and adjusting the network architecture, which is not only time-consuming and labor-intensive, but also difficult to guarantee the optimal feature extraction network every time. Furthermore, fixed, manually designed networks are difficult to flexibly optimize for the characteristics of different modal data, resulting in suboptimal feature extraction efficiency and effectiveness. This is especially true when dealing with complex and diverse multimodal data, as it makes it difficult to fully leverage the potential of each modality, thus limiting the overall performance improvement of multimodal learning.

[0005] In terms of fusion strategies, existing traditional fusion strategies are mostly fixed patterns, with common methods including feature-level fusion (such as concatenation and weighted summation) and decision-level fusion (such as voting and weighted averaging). These fixed fusion strategies have also exposed numerous problems in practical applications. On the one hand, they lack flexibility, and cannot flexibly adjust the timing and method of fusion based on factors such as the dynamic characteristics of the data, the correlation between modalities, and task requirements. This leads to limited fusion effects and an inability to fully utilize the synergistic effects of multimodal data. On the other hand, some fusion methods, such as feature-level fusion, may result in excessively high dimensionality of the fused features, thereby increasing the computational complexity of subsequent processing. This is particularly problematic when processing large-scale multimodal data, where the consumption of computing resources is particularly prominent. This, to a certain extent, limits the scalability and efficiency of multimodal learning methods in practical applications. Summary of the Invention

[0006] To address the technical issues in traditional multimodal learning, such as the strong dependency of manually designed feature extraction networks, the inefficiency of fixed fusion strategies, and the high computational cost of traditional neural architecture search, the present invention provides a multimodal automatic modeling and fusion method and device based on neural architecture search. The technical solution is as follows:

[0007] In one aspect, a multimodal automatic modeling and fusion method based on neural architecture search is provided. The method is implemented by a multimodal automatic modeling and fusion device, and the method includes:

[0008] S1. Obtain multimodal data and task types.

[0009] S2. For each modal data in the multimodal data, automatically generate the corresponding optimal unit architecture based on neural architecture search; wherein the optimal unit architecture is in the form of a directed acyclic graph.

[0010] S3. Use the optimal unit architecture as a feature extractor to extract features from each modal data separately to obtain high-dimensional features of each modal data, and store the high-dimensional features of each modal data into a shared feature cache pool.

[0011] S4. Analyze the inter-modal dependencies at the feature level based on the high-dimensional features of each modal data, generate a dynamically updated correlation heat map, and obtain a specific fusion strategy based on the correlation heat map and the fusion strategy decision maker.

[0012] S5. Build the optimal fusion network architecture based on the high-dimensional features in the shared feature cache pool and the specific fusion strategy.

[0013] S6. Based on the optimal fusion network architecture and task type, the task adaptation module automatically adjusts the output layer and loss function to obtain a network that adapts to the task requirements and completes different downstream tasks.

[0014] Optionally, for each modal data in the multimodal data, S2 automatically generates a corresponding optimal unit architecture based on neural architecture search, including:

[0015] S21. Obtain each modal data in the multimodal data and initialize a candidate operation library; the candidate operation library includes operation types and corresponding activation functions.

[0016] S22. Organize the components in the candidate operation library into a hypergraph structure to form a searchable network topology.

[0017] S23. The reinforcement learning-based Controller RNN samples subgraph structures from the hypergraph structure and instantiates the operation sequence of the sampled subgraph structure into the unit architecture of each modality.

[0018] S24. Train the subgraph structure and use a reinforcement learning search strategy to evaluate the performance of the subgraph structure, thereby generating the optimal unit architecture for each modality data.

[0019] Optionally, the subgraph structure is trained in S24, and the performance of the subgraph structure is evaluated using a reinforcement learning search strategy to generate the optimal unit architecture for each modality data, including:

[0020] The weight matrices of all operations are pre-initialized and globally shared, and only the weights of the selected operations are activated for propagation; the sampled subgraph structure is trained using the dataset of the target task, only the activated parameters are updated, and the gradients are calculated using Monte Carlo estimation; the performance of the subgraph structure is evaluated on the validation set, and the performance indicators are fed back to the controller as reward signals and normalized using sliding averages; the controller parameters are updated using a proximal policy optimization algorithm, and an entropy regularization term is added to encourage exploration. The controller parameters are updated according to the set update frequency until the maximum number of iterations is reached or the reward fluctuation is less than the set threshold, generating the optimal unit architecture for each modal data.

[0021] Optionally, S4 analyzes the inter-modal dependencies at the feature level based on the high-dimensional features of each modal data to generate a dynamically updated correlation heat map, including:

[0022] The dimension and distribution of the modal features are determined based on the high-dimensional features of each modal data, and it is judged whether the dimension of the modal features is lower than the preset threshold.

[0023] If it is not lower than the preset threshold, the distance measurement method is selected to determine the dependency between modalities and generate a dynamically updated correlation heat map.

[0024] If it is lower than the preset threshold, it is determined whether the distribution of the modal features is clear or there are significant differences in the characteristic scales; if the distribution of the modal features is clear, the statistical correlation method is selected to determine the dependency between the modalities and generate a dynamically updated correlation heat map; if there are significant differences in the characteristic scales of the distribution of the modal features, the distance measurement method is selected to determine the dependency between the modalities and generate a dynamically updated correlation heat map.

[0025] Optionally, S4 obtains a specific fusion strategy based on the correlation heat map and the fusion strategy decider, including:

[0026] Different fusion strategies are selected according to the high, medium and low correlation levels of the modal pairs in the correlation heat map. For highly correlated modal pairs, fusion is performed through shallow feature interaction to ensure that the features of the highly correlated modalities can jointly influence the model; for moderately correlated modal pairs, hybrid fusion is used to balance contributions; for low-correlation modal pairs, feature fusion is performed in depth to reduce interference between low-correlation modal pairs and avoid introducing noise.

[0027] Optionally, S5 constructs an optimal fusion network architecture based on the high-dimensional features in the shared feature cache pool and a specific fusion strategy, including:

[0028] S51, initialize the number of fusion layers , defines the maximum number of fusion layers , initialize the candidate architecture pool to store the high potential architectures screened out in each step.

[0029] S52. Layer fusion layer, using feature splicing to combine the first The output of the fusion layer is combined with the The features of the layer fusion layer are fused, and features are allowed to jump between different levels. The fused features are nonlinearly transformed through the activation function to construct the candidate architecture.

[0030] S53, for For all candidate architectures under the layer fusion layer, the sequential model based optimization SMBO algorithm is used in combination with the LSTM proxy model to predict the performance of the architecture.

[0031] S54. Sort all candidate architectures according to their performance, select multiple architectures with the highest performance as high-potential candidate architectures, and add them to the candidate architecture pool.

[0032] S55. For high-potential candidate architectures, a weight inheritance mechanism is used to initialize parameters from the historical optimal layer for training.

[0033] S56, at the maximum number of layers Under this condition, all high-potential candidate architectures in the candidate architecture pool are evaluated, and the architecture with the best performance is selected as the optimal fusion network architecture.

[0034] Optionally, in S6, the output layer and loss function are automatically adjusted by the task adaptation module according to the optimal fusion network architecture and the task type to obtain a network that adapts to the task requirements, including:

[0035] S61. Dynamically configure the output layer and loss function for the optimal fusion network architecture based on the task type.

[0036] S62. Dynamically adjust the network structure and perform two-stage training on the configured optimal fusion network architecture. In the first stage, the parameters of the task adaptation module are fixed, and the feature representation of the configured optimal fusion network architecture is optimized. In the second stage, the parameters of the optimal fusion network architecture are locked, and the task adaptation module is fine-tuned.

[0037] S63. Determine whether the downstream task type has changed. If so, unload the output layer and loss function of the current task, load the configuration of the new task, and start hot start fine-tuning. If no change occurs, monitor the performance of the optimal fusion network architecture.

[0038] On the other hand, a multimodal automatic modeling and fusion device based on neural architecture search is provided. The device is applied to a multimodal automatic modeling and fusion method based on neural architecture search, and the device includes:

[0039] The acquisition module is used to obtain multimodal data and task types.

[0040] The neural architecture search module is used to automatically generate the corresponding optimal unit architecture for each modal data in the multimodal data based on the neural architecture search; wherein the optimal unit architecture is in the form of a directed acyclic graph.

[0041] The single-modal feature extraction module is used to use the optimal unit architecture as a feature extractor to extract features from each modal data separately, obtain high-dimensional features of each modal data, and store the high-dimensional features of each modal data in a shared feature cache pool.

[0042] The modal correlation calculation module is used to analyze the inter-modal dependencies at the feature level based on the high-dimensional features of each modal data, generate a dynamically updated correlation heat map, and obtain a specific fusion strategy based on the correlation heat map and the fusion strategy decider.

[0043] The dynamic fusion network module is used to build the optimal fusion network architecture based on the high-dimensional features in the shared feature cache pool and specific fusion strategies.

[0044] The task adaptation module is used to automatically adjust the output layer and loss function according to the optimal fusion network architecture and task type, obtain a network that adapts to task requirements, and complete different downstream tasks.

[0045] On the other hand, a multimodal automatic modeling and fusion device is provided, which includes: a processor; a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned multimodal automatic modeling and fusion methods based on neural architecture search is implemented.

[0046] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned multimodal automatic modeling and fusion methods based on neural architecture search.

[0047] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0048] In the present invention, efficient and low-cost end-to-end multimodal automatic modeling is achieved by jointly optimizing single-modal feature extraction and multimodal feature fusion through automated architecture search. The core of this method is to use a neural architecture search mechanism based on reinforcement learning search strategy to automatically generate lightweight network units to build an efficient feature extraction network, while reducing training overhead through a parameter sharing mechanism. In the multimodal fusion stage, by quantitatively analyzing the correlation between modalities, the optimal fusion time is dynamically decided, breaking through the limitations of traditional fixed strategies. The present invention combines progressive search with proxy model prediction. Progressive search efficiently screens out the optimal fusion architecture by gradually increasing the complexity of the search space; proxy model prediction technology quickly evaluates the performance of candidate architectures by constructing lightweight proxy models, further accelerating the automatic search process. In addition, the present invention can adaptively select loss functions according to the task type and support multiple task requirements. The innovation of the present invention lies in: proposing a dynamic correlation-guided fusion mechanism to achieve joint search and global optimization of single-modal neural architecture search and multimodal fusion architecture, and complete efficient automatic modeling. This method is a general multimodal solution that can be widely applied to various multimodal application scenarios. It has high generalization, strong noise robustness and cross-domain migration capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a flow chart of a multimodal automatic modeling and fusion method based on neural architecture search provided by an embodiment of the present invention;

[0051] Figure 2 This is a framework diagram of a multimodal automatic modeling and fusion method based on neural architecture joint search provided by an embodiment of the present invention;

[0052] Figure 3 is a flow chart of a neural architecture search module provided by an embodiment of the present invention;

[0053] Figure 4 is a multimodal network fusion flow chart provided by an embodiment of the present invention;

[0054] Figure 5 is a flowchart of a task adaptation module provided by an embodiment of the present invention;

[0055] Figure 6 This is a flow chart of a multimodal automatic modeling and fusion method based on neural architecture search provided by an embodiment of the present invention;

[0056] Figure 7 This is a block diagram of a multimodal automatic modeling and fusion device based on neural architecture search provided by an embodiment of the present invention;

[0057] Figure 8 It is a structural diagram of a multimodal automatic modeling and fusion device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0059] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0060] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0061] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0062] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0063] The embodiment of the present invention provides a multimodal automatic modeling and fusion method based on neural architecture search, which can be implemented by a multimodal automatic modeling and fusion device, which can be a terminal or a server. Figure 1 The flowchart of the multimodal automatic modeling and fusion method based on neural architecture search is shown. The processing flow of the method may include the following steps:

[0064] S1. Obtain multimodal data and task types.

[0065] In one feasible implementation, the present invention can be applied to the fields of computer science, as well as fields such as the Internet of Things, the Industrial Internet of Things, transportation, and healthcare. Furthermore, it can be applied to the field of multimodal data fusion, particularly in complex scenarios requiring efficient processing and fusion of multiple modal data (such as text, images, audio, and video).

[0066] Specifically, in the field of intelligent medical care, the system automatically constructs a 3D convolutional network through NAS to process CT image sequences, and uses the Transformer model to parse electronic medical record texts. It realizes feature fusion of the lesion area based on the dynamically generated correlation heat map, providing accurate assistance for clinical diagnosis.

[0067] In industrial IoT scenarios, cross-modal correlation analysis is performed on equipment vibration spectra, infrared thermal imaging, and operation and maintenance logs. Vibration signals are processed using a one-dimensional convolutional network automatically optimized by NAS, and thermal imaging features are extracted using a visual attention mechanism to build an end-to-end equipment health status prediction model.

[0068] In the field of intelligent transportation, NAS can automatically generate a multi-scale feature extraction network to process vehicle-mounted camera data, while searching for the optimal point cloud sparse convolutional architecture to parse lidar information, and improve environmental perception capabilities through automatically optimized fusion strategies.

[0069] In smart city management, a spatiotemporal attention mechanism can be introduced to fuse surveillance video streams, social media texts, and traffic flow time series data to build an urban event early warning system.

[0070] These application solutions demonstrate the unique advantages of this invention in processing multi-faceted heterogeneous information such as text, images, audio, video, and sensor data, and provide customized multimodal solutions for various industries through automated neural architecture search and intelligent fusion strategies.

[0071] like Figure 2 As shown, the present invention proposes a multimodal automatic modeling and fusion method based on neural architecture joint search, which realizes efficient modeling from single-modal feature extraction to multimodal fusion through a fully automated process, and provides an end-to-end solution for adaptive interaction and efficient joint modeling of cross-modal data.

[0072] S2. For each modal data in the multimodal data, automatically generate the corresponding optimal unit architecture based on neural architecture search; wherein the optimal unit architecture is in the form of a directed acyclic graph.

[0073] In one feasible implementation, each unimodal unit architecture consists of a dynamic DAG (Directed Acyclic Graph) generated by a controller, where nodes contain candidate operations and activation functions. The controller is optimized through a reinforcement learning strategy to generate a high-reward structure, and a parameter sharing mechanism is adopted to reduce computational costs. Monte Carlo gradient estimation is used to update shared weights to achieve automatic modeling of the unimodal feature extraction network.

[0074] Optionally, the above step S2 may include the following steps S21-S24:

[0075] S21. Obtain each modal data in the multimodal data and initialize a candidate operation library; the candidate operation library includes operation types and corresponding activation functions.

[0076] S22. Organize the components in the candidate operation library into a hypergraph structure to form a searchable network topology.

[0077] S23. The reinforcement learning-based Controller RNN samples subgraph structures from the hypergraph structure and instantiates the operation sequence of the sampled subgraph structure into the unit architecture of each modality.

[0078] S24. Train the subgraph structure and use a reinforcement learning search strategy to evaluate the performance of the subgraph structure, thereby generating the optimal unit architecture for each modality data.

[0079] Specifically, the above step S24 may include:

[0080] The weight matrices of all operations are pre-initialized and globally shared, and only the weights of the selected operations are activated for propagation; the sampled subgraph structure is trained using the dataset of the target task, only the activated parameters are updated, and the gradients are calculated using Monte Carlo estimation; the performance of the subgraph structure is evaluated on the validation set, and the performance indicators are fed back to the controller as reward signals and normalized using sliding averages; the controller parameters are updated using a proximal policy optimization algorithm, and an entropy regularization term is added to encourage exploration. The controller parameters are updated according to the set update frequency until the maximum number of iterations is reached or the reward fluctuation is less than the set threshold, generating the optimal unit architecture for each modal data.

[0081] In one feasible implementation, the neural architecture search process first defines the search space and specifies the operation types in the candidate set, including different candidate operations, activation functions, and other operations such as identity mapping and zeroing. The unit structure is organized as a directed acyclic graph, with nodes representing operations and edges representing data flow. Input and output nodes are clearly defined, and the number of intermediate nodes is set within a certain range. The controller network uses a two-layer LSTM (Long Short-Term Memory) architecture, with the weight matrix and bias randomly initialized to a uniform distribution. The controller input is an encoded representation of the sampled operation sequence, and the output is a probability distribution of the next operation. During the training and validation phases, the operation sequence sampled by the controller is used to construct a DAG structure to generate a specific unit architecture. The weight matrices of all possible operations are pre-initialized and globally shared, with only the weights of the selected operations being activated for propagation. The sampled sub-architecture is trained using the target task dataset, updating only the activated shared parameters and computing gradients using Monte Carlo estimation. The sub-architecture performance is evaluated on the validation set, and the performance metrics are fed back to the controller as reward signals, with sliding average normalization applied. The controller parameters are updated using the Proximal Policy Optimization (PPO) algorithm, with an entropy regularization term added to encourage exploration. The controller parameters are updated according to the set update frequency. The search termination criteria include reaching the maximum number of iterations or reward fluctuations falling below a set threshold. During the search, the performance metrics of each sampled sub-architecture are recorded, and at the end of the search, the unit architecture with the best performance is selected as the final architecture. If the performance of a sub-architecture is significantly better than that of other sub-architectures and the performance improvement is very small over multiple consecutive iterations, the search can be terminated early. Next, the modal correlation calculation module dynamically determines the timing and method of fusion to achieve effective fusion of multimodal features.

[0082] Neural architecture search process, such as Figure 3 The specific steps are as follows:

[0083] ① Input the raw data of each modality into the system and initialize the candidate operation library, including possible operation types and their corresponding activation functions;

[0084] ② Organize the components in the operation library into a hypergraph structure to form a searchable network topology.

[0085] ③ The reinforcement learning-based Controller RNN samples subgraph structures from the hypergraph and instantiates the sampled operation sequence into a specific network structure (each modal unit architecture);

[0086] ④ Train the subgraph and evaluate its performance, feed back optimization signals, use reinforcement learning search strategies, use the unit structure as the action space, and evaluate the performance of different structures through reward mechanisms;

[0087] ⑤ Record the accuracy of the validation set and feed it back to the Controller RNN as a reward signal to evaluate the performance of different unit architectures and select the architecture with the best performance;

[0088] ⑥ After searching for the optimal unit structure, use the optimized unit architecture for feature extraction.

[0089] This paper uses neural architecture search technology to automatically search for the optimal single-modal network unit, using it as a feature extractor for multimodal feature fusion. It then automatically selects and adjusts the multimodal feature fusion architecture, leveraging the complementarity and correlation between modalities to maximize information integration and improve the performance and efficiency of multimodal data processing. Through intelligent architecture search, the method can adaptively fuse different data sets and tasks, avoiding the complexity of manual design and effectively improving the accuracy and robustness of multimodal learning systems in practical applications.

[0090] Furthermore, the present invention proposes a neural architecture search mechanism based on directed acyclic graphs, constructs a search space of shareable sub-modules, and contains multiple ordered connected neural network operation nodes. Each node selects the optimal operation from a preset operation candidate set and defines the feature flow direction through directed edges. A controller network is used to sample the optimal architecture in the search space through a reinforcement learning strategy, and all sampled sub-architectures share weight parameters through a parameter sharing mechanism. The controller network and sub-architecture parameters are alternately trained to optimize the overall model, and the optimal lightweight network unit is automatically generated.

[0091] S3. Use the optimal unit architecture as a feature extractor to extract features from each modal data separately to obtain high-dimensional features of each modal data, and store the high-dimensional features of each modal data into a shared feature cache pool.

[0092] In one possible implementation, during the feature extraction and fusion configuration phase, the optimal unit architecture found is used as a feature extractor to extract features from the input data, extracting features from different layers as needed. The extracted features can be represented as fixed-length vectors, which are then normalized to ensure comparability between different modalities.

[0093] The present invention adopts automatically generated lightweight network units as the core feature extractor, which can adapt to the feature extraction requirements of different modal data, maximize the feature expression ability while maintaining computational efficiency, and form a complete feature extraction network by cascading or parallel connection of multiple units to realize multi-level feature abstraction and representation learning of input data.

[0094] S4. Analyze the inter-modal dependencies at the feature level based on the high-dimensional features of each modal data, generate a dynamically updated correlation heat map, and obtain a specific fusion strategy based on the correlation heat map and the fusion strategy decision maker.

[0095] Optionally, S4 analyzes the inter-modal dependencies at the feature level based on the high-dimensional features of each modal data to generate a dynamically updated correlation heat map, including:

[0096] The dimension and distribution of the modal features are determined based on the high-dimensional features of each modal data, and it is judged whether the dimension of the modal features is lower than the preset threshold.

[0097] If it is not lower than the preset threshold, the distance measurement method is selected to determine the dependency between modalities and generate a dynamically updated correlation heat map.

[0098] If it is lower than the preset threshold, it is determined whether the distribution of the modal features is clear or there are significant differences in the characteristic scales; if the distribution of the modal features is clear, the statistical correlation method is selected to determine the dependency between the modalities and generate a dynamically updated correlation heat map; if there are significant differences in the characteristic scales of the distribution of the modal features, the distance measurement method is selected to determine the dependency between the modalities and generate a dynamically updated correlation heat map.

[0099] Optionally, S4 obtains a specific fusion strategy based on the correlation heat map and the fusion strategy decider, including:

[0100] Different fusion strategies are selected according to the high, medium and low correlation levels of the modal pairs in the correlation heat map. For highly correlated modal pairs, fusion is performed through shallow feature interaction to ensure that the features of the highly correlated modalities can jointly influence the model; for moderately correlated modal pairs, hybrid fusion is used to balance contributions; for low-correlation modal pairs, feature fusion is performed in depth to reduce interference between low-correlation modal pairs and avoid introducing noise.

[0101] In one feasible implementation, the modal correlation calculation module plays a key role in the multimodal fusion system. Its main processes include feature extraction, caching, correlation calculation, and fusion strategy adjustment. The system automatically extracts features from each modality through a lightweight unit and stores these features in a feature cache pool. Each modality's features are assigned to a corresponding cache space for subsequent use. The system then activates the modal correlation calculation module to calculate the dependencies between the modalities. First, the dimensionality and distribution of the modal features are determined. If the modal features have low dimensionality and a clear distribution, a statistical correlation method is selected. This method uses mutual information or a covariance matrix to quantify the dependencies between the modalities, visually reflecting the overall degree of correlation between the modalities. If the modal features have high dimensionality or significant differences in feature scale, a distance metric is selected. Euclidean distance, cosine similarity, Manhattan distance, and other methods are used to measure the similarity or difference between feature vectors. The correlation results are displayed as a heat map, and different fusion strategies are selected based on the high, medium, or low correlation levels in the heat map. For highly correlated modal pairs, early fusion is used to capture strong dependencies, and fusion is performed through shallow feature interactions (such as feature concatenation or weighted summation) to ensure that the features of highly correlated modalities can jointly influence the model. For moderately correlated modalities, hybrid fusion is used to balance contributions, including multi-layer feature interactions, such as mid-layer feature concatenation and weighted attention mechanisms. For low-correlation modalities, late fusion is selected, that is, feature fusion is performed at a deep level to reduce interference between low-correlation modalities and avoid the introduction of noise. In addition, the system regularly recalculates the correlation between modalities based on changes in feature representation, adjusts the fusion strategy in real time, and dynamically updates the heat map to ensure optimal performance. If you are unsure which method is more appropriate, you can calculate both indicators simultaneously and conduct a comprehensive analysis based on experimental results to ensure that the selected fusion strategy is the best.

[0102] The present invention evaluates fusion benefits in real time based on the calculation of mutual information between modalities. By dynamically calculating the correlation between modalities and analyzing task requirements, the optimal fusion strategy is intelligently selected. An early fusion strategy is adopted for high-correlation modalities to fully exploit complementary information; for medium-correlation modalities, mid-level abstract features are selectively fused to balance modality-specific retention and cross-modal association modeling; and a late fusion strategy is adopted for low-correlation modalities to reduce information redundancy. By evaluating the dependencies between modalities in real time and dynamically adjusting the fusion method, the optimal information integration efficiency is ensured for different tasks and datasets, significantly reducing computational overhead while improving model accuracy.

[0103] The dynamic correlation-guided fusion timing decision method achieves adaptive fusion by quantifying the strength of inter-modal interactions. This method calculates the correlation of modal features based on normalized mutual information and dynamically selects an early or late fusion strategy. High-correlation modalities are prioritized for connecting shallow features to capture underlying relationships, while low-correlation modalities are optimized for deep feature interaction to avoid noise interference. This significantly improves cross-modal semantic alignment and noise robustness.

[0104] S5. Build the optimal fusion network architecture based on the high-dimensional features in the shared feature cache pool and the specific fusion strategy.

[0105] Optionally, the above step S5 may include the following steps S51-S56:

[0106] S51, initialize the number of fusion layers , defines the maximum number of fusion layers , initialize the candidate architecture pool to store the high potential architectures screened out in each step.

[0107] S52. Layer fusion layer, using feature splicing to combine the first The output of the fusion layer is combined with the The features of the layer fusion layer are fused, and features are allowed to jump between different levels. The fused features are nonlinearly transformed through the activation function to construct the candidate architecture.

[0108] S53, for For all candidate architectures under the layer fusion layer, the sequential model based optimization SMBO algorithm is used in combination with the LSTM proxy model to predict the performance of the architecture.

[0109] S54. Sort all candidate architectures according to their performance, select multiple architectures with the highest performance as high-potential candidate architectures, and add them to the candidate architecture pool.

[0110] S55. For high-potential candidate architectures, a weight inheritance mechanism is used to initialize parameters from the historical optimal layer for training.

[0111] S56, at the maximum number of layers Under this condition, all high-potential candidate architectures in the candidate architecture pool are evaluated, and the architecture with the best performance is selected as the optimal fusion network architecture.

[0112] In one feasible implementation, after calculating the correlation between modalities, the fusion operation determines how to merge features based on the correlation. Different fusion strategies use modal correlation to perform weighted fusion or adjust the way information flows. At each fusion layer, features from different modalities are merged using feature splicing, and cross-modal information exchange can be enhanced through weighted splicing, addition, and other methods. Each fusion layer not only fuses the features of the current layer with the output of the previous layer, but also performs nonlinear transformations through activation functions to obtain a comprehensive multimodal representation. The fusion operation of each layer interacts the output of the previous layer with the multimodal features of the current layer, thereby achieving more flexible message passing. Skip connections can directly transmit important features to avoid feature loss or degradation. After the output of each fusion layer, an activation function is used to introduce nonlinear transformations to enhance the network's expressive power. The multimodal feature fusion configuration provides a flexible way to process data from different modalities. By configuring the matrix to define the strategy for selecting specific features from each modality for fusion, as well as the nonlinear activation function used in each fusion layer, the feature processing method can be dynamically adjusted according to task requirements.

[0113] The unimodal neural architecture search unit is used as a pre-trained multi-layer feature extractor, assuming that modes, each with a function , is the input data, It is a neural network composed of multiple sub-functions. A neural network realizes the fusion of multimodal data. Each fusion layer Output It is defined by the following formula:

[0114] (1)

[0115] in, Indicates the The output of the fusion layer, It is The search targets of the fusion layers represent a certain intermediate hidden feature in each modality and the activation function used. The intermediate layer features of each modality are extracted through the optimal unit architecture generated by the neural architecture search module. is the output of the previous fusion layer. For the first fusion layer No . represents the candidate fusion function, Indicates the fusion operation selection index. is the applied nonlinear activation function, which is given by the variable Sure.

[0116] Furthermore, the dynamic fusion network module builds a cross-modal interaction architecture through a progressive search strategy, and achieves efficient and flexible multimodal feature fusion through a phased training process. The entire training process consists of two key stages: search and final training. In the search stage, the algorithm starts from the simplest fusion architecture, which has only one fusion layer ( ) and gradually increase the number of fusion layers to explore more complex architectures. In each iteration, the system samples and trains based on the surrogate model. The candidate architecture is evaluated, and the proxy model predicts the performance of the new architecture by learning historical architecture performance data. In order to balance exploration and utilization of existing knowledge, the system introduces a temperature parameter The initial high temperature value makes the sampling more random, and the temperature is gradually reduced as the search progresses, making the sampling more dependent on the prediction results of the proxy model. At the same time, the system uses a weight sharing mechanism to reduce the computational cost, allowing sampling architectures with similar structures to share the weight matrix of the same layer. The performance of the sampling architecture is recorded at each iteration, and the prediction ability of the proxy model is continuously optimized. After several iterations, the search phase ends and the system selects the best performing The candidate architectures enter the final training phase. In this phase, these candidate architectures receive more thorough training, including increasing the number of training cycles and expanding the size of the weight matrix to ensure that the architecture's performance is fully exploited. Unlike the search phase, the final training phase not only trains the weights of the fusion layer but also fine-tunes the weights of the feature extractor. By using a multi-task loss function, the output of each modality and the fusion output are simultaneously optimized to ensure optimal performance of the entire network. Finally, the performance of all candidate architectures is evaluated on the validation set, and the best-performing architecture is selected as the final multimodal fusion model output for subsequent tasks. This training method, by combining intelligent search strategies with thorough training, can not only find the optimal architecture in a vast search space, but also ensure the model's performance in real-world tasks.

[0117] The workflow of the multimodal fusion network is as follows: Figure 4 As shown:

[0118] Each step of the flowchart is described in detail:

[0119] ① Initialize the number of fusion layers. Set the initial parameters and Initialized to 1, indicating starting from a single layer fusion. At the same time, define the maximum number of layers , indicating that it is gradually expanded to a maximum Layer fusion. Initialize the candidate architecture pool and create an empty candidate architecture pool to store the high potential architectures screened out at each step.

[0120] ②Build a cross-modal interaction architecture. Under the hood, a fusion layer is constructed to support cross-modal jump connections and reuse of historical layer features. Each fusion layer can receive features from different modalities and allow jump connections between features at different levels, while reusing features from historical layers.

[0121] ③ Performance prediction. Use SMBO algorithm: Use the Sequential Model Based Optimization (SMBO) algorithm in combination with the LSTM proxy model to predict the performance of the untrained architecture. The SMBO algorithm sequentially evaluates the performance of candidate architectures and uses a proxy model (such as LSTM) to predict the performance of the untrained architecture. The LSTM proxy model learns the performance pattern of the architecture based on historical data. The input is the feature representation of the architecture, and the output is the predicted performance indicators such as accuracy, loss, etc. For the current number of fusion layers For all candidate architectures under , their performance is predicted using the LSTM proxy model.

[0122] ④ Screen high-potential candidate architectures. Rank all candidate architectures based on the performance predicted by the LSTM proxy model and select the top-3 architectures with the highest performance as high-potential candidate architectures. These architectures are considered to have good performance potential in subsequent training. The screened high-potential candidate architectures are added to the candidate architecture pool for subsequent training and evaluation.

[0123] ⑤ For selected high-potential candidate architectures, a weight inheritance mechanism is used to initialize parameters from the historically optimal layer. The parameters of the previously trained optimal layer are used as the initial parameters for the current architecture, reducing redundant computation and improving training efficiency. When initializing parameters, the parameters of the historically optimal layer are reused to avoid starting with random initialization, thereby accelerating the training process.

[0124] ⑥ Output the optimal fusion architecture and evaluate the final candidate architecture. Next, all architectures in the candidate architecture pool are finally evaluated and the architecture with the best performance is selected for subsequent multimodal data processing tasks.

[0125] In this paper, a progressive search strategy optimizes search efficiency by gradually expanding the complexity of the fusion network. Starting from a single-layer fusion architecture, the number of layers is gradually increased while inheriting the historically optimal parameter weights. In combination with an early stopping mechanism, redundant calculations are avoided, improving the efficiency of automatic modeling and fusion architecture search.

[0126] End-to-end training uses a two-stage optimization strategy to achieve global parameter coordination. The first stage freezes the single-modal network to quickly converge the fusion layer. The second stage jointly fine-tunes all parameters and introduces a dynamic learning rate and weight decay mechanism to support flexible adaptation to multiple tasks.

[0127] The present invention constructs a dynamically configurable multimodal feature fusion network architecture, automatically designs flexible feature fusion strategies based on the results of inter-modal correlation calculations, and creates a candidate architecture pool covering a variety of possible fusion configurations, such as feature splicing, weighted summation, etc. The correlation analysis results are used to dynamically select the fusion configuration that best suits the current data and task. Then, a feature fusion module is designed that can perform corresponding operations according to different fusion strategies and is easy to expand to adapt to new fusion strategies. Cross-layer jump connections are used in the network to enhance feature interaction, nonlinear activation functions are integrated to increase the expressive power of the network, and a complete multimodal fusion network architecture is constructed that includes an automatic feature extraction layer, an intelligent decision-making and fusion layer. The network is trained through end-to-end training and multi-task learning strategies, and its performance is evaluated on a validation set. The fusion strategy and network structure are adjusted based on feedback to form a dynamic, configurable multimodal automatic modeling and fusion system that is adaptive to different modal data and task requirements.

[0128] S6. Based on the optimal fusion network architecture and task type, the task adaptation module automatically adjusts the output layer and loss function to obtain a network that adapts to the task requirements and completes different downstream tasks.

[0129] Optionally, the above step S6 may include the following steps S61-S63:

[0130] S61. Dynamically configure the output layer and loss function for the optimal fusion network architecture based on the task type.

[0131] S62. Dynamically adjust the network structure and perform two-stage training on the configured optimal fusion network architecture. In the first stage, the parameters of the task adaptation module are fixed, and the feature representation of the configured optimal fusion network architecture is optimized. In the second stage, the parameters of the optimal fusion network architecture are locked, and the task adaptation module is fine-tuned.

[0132] S63. Determine whether the downstream task type has changed. If so, unload the output layer and loss function of the current task, load the configuration of the new task, and start hot start fine-tuning. If no change occurs, monitor the performance of the optimal fusion network architecture.

[0133] In one feasible implementation, the task adaptation module is responsible for automatically adjusting the output layer and loss function based on the optimal fusion architecture output by the multimodal fusion network to adapt to the needs of different downstream tasks. This module can select the most appropriate configuration according to the different types of tasks and perform two-stage training optimization. The workflow of the task adaptation module is as follows: Figure 5 As shown:

[0134] ① Load the best fusion architecture obtained in the architecture search phase.

[0135] ② Dynamically configure the output layer and loss function based on the task type. For example, for classification tasks, use a Softmax or Sigmoid output layer with label smoothed cross entropy loss, for regression tasks, use a linear output layer with adaptive Huber loss, and for generation tasks, configure a Transformer decoder with a combination of adversarial loss and L1 reconstruction loss.

[0136] ③ Perform dynamic network structure adjustment and two-stage training. In the first stage, the task adaptation module parameters are fixed, focusing on optimizing the fusion network feature representation. In the second stage, the fusion network parameters are locked and the focus is on fine-tuning the task adaptation module. This two-stage separate training approach ensures the independence of feature learning while improving training efficiency.

[0137] ④ Implement real-time task switching and performance monitoring. When a change in downstream task type is detected, the module immediately unloads the output layer and loss function of the current task, loads the configuration of the new task, and initiates hot-start fine-tuning. This process is based on historical fusion parameters and eliminates the need for re-pre-training. Simultaneously, the system monitors metrics such as loss value and accuracy in real time to ensure system stability. When performance fluctuations exceed thresholds, an alarm and version rollback mechanism are automatically triggered.

[0138] like Figure 6 As shown, the present invention jointly optimizes unimodal feature extraction and multimodal fusion strategies through automated architecture search. The system first automatically generates the optimal searchable lightweight unimodal unit structure for different modal data to achieve efficient unimodal automatic modeling. After searching for the optimal unit structure, the optimized unit architecture is used for feature extraction, and the extracted unimodal features are fused and configured. The fusion configuration combines the output of the previous fusion layer and a nonlinear activation function to form a multimodal fusion search space. A progressive search strategy is adopted, combined with the SMBO algorithm, to gradually expand the complexity from a simple architecture, and the LSTM proxy model is used to predict the performance of the untrained architecture to reduce the search time. Finally, the loss function and output layer are adaptively selected according to the task type to achieve end-to-end multimodal fusion network model training and efficient fusion of multimodal features.

[0139] In an embodiment of the present invention, efficient and low-cost end-to-end multimodal automatic modeling is achieved by jointly optimizing single-modal feature extraction and multimodal feature fusion through automated architecture search. The core of this method is to automatically generate lightweight network units to construct an efficient feature extraction network using a neural architecture search mechanism based on a reinforcement learning search strategy, while reducing training overhead through a parameter sharing mechanism. In the multimodal fusion stage, the optimal fusion timing is dynamically determined by quantitatively analyzing the correlation between modalities, breaking through the limitations of traditional fixed strategies. The present invention combines progressive search with proxy model prediction. Progressive search effectively screens the optimal fusion architecture by gradually increasing the complexity of the search space; proxy model prediction technology quickly evaluates the performance of candidate architectures by constructing lightweight proxy models, further accelerating the automatic search process. In addition, the present invention can adaptively select loss functions based on the task type and support multiple task requirements. The innovation of the present invention lies in: proposing a dynamic correlation-guided fusion mechanism to achieve joint search and global optimization of single-modal neural architecture search and multimodal fusion architecture, completing efficient automatic modeling. This method is a general multimodal solution that can be widely applied to various multimodal application scenarios. It has high generalization, strong noise robustness and cross-domain migration capabilities.

[0140] Figure 7 This is a block diagram of a multimodal automatic modeling and fusion device based on neural architecture search according to an exemplary embodiment. The device is used for a multimodal automatic modeling and fusion method based on neural architecture search. Figure 7 The device includes an acquisition module 310, a neural architecture search module 320, a single-modal feature extraction module 330, a modality correlation calculation module 340, a dynamic fusion network module 350, and a task adaptation module 360.

[0141] The acquisition module 310 is used to acquire multimodal data and task types.

[0142] The neural architecture search module 320 is used to automatically generate a corresponding optimal unit architecture for each modal data in the multimodal data based on the neural architecture search; wherein the optimal unit architecture is in the form of a directed acyclic graph.

[0143] The single-modal feature extraction module 330 is used to use the optimal unit architecture as a feature extractor to extract features from each modal data separately, obtain high-dimensional features of each modal data, and store the high-dimensional features of each modal data in a shared feature buffer pool.

[0144] The modality correlation calculation module 340 is used to analyze the inter-modality dependency at the feature level based on the high-dimensional features of each modality data, generate a dynamically updated correlation heat map, and obtain a specific fusion strategy based on the correlation heat map and the fusion strategy decider.

[0145] The dynamic fusion network module 350 is used to construct an optimal fusion network architecture based on the high-dimensional features in the shared feature buffer pool and a specific fusion strategy.

[0146] The task adaptation module 360 ​​is used to automatically adjust the output layer and loss function according to the optimal fusion network architecture and task type to obtain a network that adapts to the task requirements and completes different downstream tasks.

[0147] In one feasible implementation, the neural architecture search module automatically generates lightweight unit structures based on neural architecture search. Each unit contains searchable neural network operations. A reinforcement learning controller dynamically selects the optimal operation sequence from a candidate set of operations, and trains the candidate architecture using a weight sharing mechanism. Differentiated search constraints are designed for different modal characteristics, and the output optimal unit architecture is stored as a directed acyclic graph, allowing for flexible stacking and expansion.

[0148] Single-modal feature extraction module: The optimal unit found in the search is used as the feature extractor. These optimized units have better feature extraction capabilities and model performance, and can be flexibly stacked to build larger or smaller networks, further enhancing feature extraction capabilities and learning more complex feature representations. Each modality's data is processed separately by its corresponding unit network, and the extracted high-dimensional features are stored in a shared feature buffer pool, providing standardized input for subsequent fusion.

[0149] Modal Correlation Calculation Module: After each modal data is extracted using automatically generated lightweight units and stored in the feature cache, the system initiates the Modal Correlation Calculation Module. This module quantitatively analyzes inter-modal dependencies at the feature level using statistical correlation methods or distance metrics, generating a dynamically updated correlation heat map. The results directly drive the fusion strategy decision maker to select the optimal fusion method. Periodic recalculation adapts to the dynamic evolution of feature representation capabilities, ensuring that modal correlation calculations are performed at the optimal feature quality stage. This module is also decoupled from the architecture search process to reduce complexity.

[0150] Dynamic Fusion Network Module: This module receives multimodal features generated by the unimodal feature extraction module from the feature cache pool and initializes it with a specific fusion strategy provided by the modal correlation calculation module to construct a minimum viable architecture consisting of basic units and a simple fusion layer. During the progressive expansion process, the system dynamically generates a pool of candidate architectures through feature fusion configuration steps, inserts high-order components such as the attention mechanism, and uses Sequential Model Optimization (SMBO) for evaluation. Only the top 20% of potential architectures are actually trained. The search is terminated when the performance improvement is less than 1% after three consecutive iterations or when the preset complexity threshold is reached. The module ultimately outputs the optimal fusion network architecture that interfaces with the task adaptation module. This module achieves efficient feature interaction through a weight sharing mechanism and dynamic policy drive, ensuring end-to-end collaborative optimization from feature extraction to result output.

[0151] Task Adaptation Module: This module automatically adjusts the output layer and loss function based on the optimal fusion architecture output by the multimodal fusion network to meet the needs of different downstream tasks. The most appropriate configuration is selected based on the different task types, and two-stage training optimization is performed. In the first stage, the unimodal network parameters are frozen to quickly converge the fusion layer. In the second stage, all parameters are jointly fine-tuned, and generalization capabilities are improved through dynamic learning rates and weight decay. The end-to-end joint optimization mechanism integrates the network performance gradient and feeds it back to the unimodal unit generation process through the NAS controller, achieving coordinated optimization of feature extraction and fusion strategies.

[0152] In an embodiment of the present invention, efficient and low-cost end-to-end multimodal automatic modeling is achieved by jointly optimizing single-modal feature extraction and multimodal feature fusion through automated architecture search. The core of this method is to automatically generate lightweight network units to construct an efficient feature extraction network using a neural architecture search mechanism based on a reinforcement learning search strategy, while reducing training overhead through a parameter sharing mechanism. In the multimodal fusion stage, the optimal fusion timing is dynamically determined by quantitatively analyzing the correlation between modalities, breaking through the limitations of traditional fixed strategies. The present invention combines progressive search with proxy model prediction. Progressive search effectively screens the optimal fusion architecture by gradually increasing the complexity of the search space; proxy model prediction technology quickly evaluates the performance of candidate architectures by constructing lightweight proxy models, further accelerating the automatic search process. In addition, the present invention can adaptively select loss functions based on the task type and support multiple task requirements. The innovation of the present invention lies in: proposing a dynamic correlation-guided fusion mechanism to achieve joint search and global optimization of single-modal neural architecture search and multimodal fusion architecture, completing efficient automatic modeling. This method is a general multimodal solution that can be widely applied to various multimodal application scenarios. It has high generalization, strong noise robustness and cross-domain migration capabilities.

[0153] Figure 8Schematic diagram of a multimodal automatic modeling and fusion device provided by an embodiment of the present invention, such as Figure 8 As shown, the multimodal automatic modeling and fusion device may include the above Figure 7 The multimodal automatic modeling and fusion device based on neural architecture search is shown. Optionally, the multimodal automatic modeling and fusion device 410 may include a first processor 2001.

[0154] Optionally, the multimodal automatic modeling and fusion device 410 may further include a memory 2002 and a transceiver 2003 .

[0155] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0156] The following combination Figure 8 The components of the multimodal automatic modeling and fusion device 410 are described in detail:

[0157] The first processor 2001 is the control center of the multimodal automatic modeling and fusion device 410 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0158] Optionally, the first processor 2001 may execute various functions of the multimodal automatic modeling and fusion device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0159] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.

[0160] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0161] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A multimodal automatic modeling and fusion method based on neural architecture search, characterized in that: The method comprises: S1, obtain multimodal data and task types; S2. For each modal data in the multimodal data, automatically generate a corresponding optimal unit architecture based on neural architecture search; wherein the optimal unit architecture is in the form of a directed acyclic graph; S3. Using the optimal unit architecture as a feature extractor, extract features from each modal data to obtain high-dimensional features of each modal data, and storing the high-dimensional features of each modal data in a shared feature buffer pool; S4. Analyzing the inter-modal dependency at the feature level based on the high-dimensional features of each modal data, generating a dynamically updated correlation heat map, and obtaining a specific fusion strategy based on the correlation heat map and a fusion strategy decider; S5. Build an optimal fusion network architecture based on the high-dimensional features in the shared feature cache pool and a specific fusion strategy; S6. Based on the optimal fusion network architecture and task type, the task adaptation module automatically adjusts the output layer and loss function to obtain a network that adapts to the task requirements and completes different downstream tasks.

2. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 1 is characterized in that: The S2 automatically generates the corresponding optimal unit architecture for each modal data in the multimodal data based on the neural architecture search, including: S21. Obtain each modal data in the multimodal data and initialize a candidate operation library; the candidate operation library includes operation types and corresponding activation functions; S22. Organize the components in the candidate operation library into a hypergraph structure to form a searchable network topology; S23. The reinforcement learning-based Controller RNN samples subgraph structures from the hypergraph structure and instantiates the operation sequence of the sampled subgraph structure into a unit architecture for each modality; S24. Train the subgraph structure and use a reinforcement learning search strategy to evaluate the performance of the subgraph structure, thereby generating the optimal unit architecture for each modality data.

3. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 2 is characterized in that: The subgraph structure is trained in S24, and the performance of the subgraph structure is evaluated using a reinforcement learning search strategy to generate the optimal unit architecture for each modality data, including: The weight matrices of all operations are pre-initialized and globally shared, and only the weights of the selected operations are activated for propagation; the sampled subgraph structure is trained using the dataset of the target task, only the activated parameters are updated, and the gradients are calculated using Monte Carlo estimation; the performance of the subgraph structure is evaluated on the validation set, and the performance indicators are fed back to the controller as reward signals and normalized using sliding averages; the controller parameters are updated using a proximal policy optimization algorithm, and an entropy regularization term is added to encourage exploration. The controller parameters are updated according to the set update frequency until the maximum number of iterations is reached or the reward fluctuation is less than the set threshold, generating the optimal unit architecture for each modal data.

4. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 1 is characterized in that: The step S4 of analyzing the inter-modal dependency at the feature level based on the high-dimensional features of each modal data to generate a dynamically updated correlation heat map includes: Determining the dimension and distribution of the modal features based on the high-dimensional features of each modal data, and judging whether the dimension of the modal features is lower than a preset threshold; If it is not lower than the preset threshold, a distance measurement method is selected to determine the dependency between modalities and generate a dynamically updated correlation heat map; If it is lower than the preset threshold, it is determined whether the distribution of the modal features is clear or there are significant differences in the characteristic scales; if the distribution of the modal features is clear, the statistical correlation method is selected to determine the dependency between the modalities and generate a dynamically updated correlation heat map; if there are significant differences in the characteristic scales of the distribution of the modal features, the distance measurement method is selected to determine the dependency between the modalities and generate a dynamically updated correlation heat map.

5. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 1 is characterized in that: The specific fusion strategy is obtained according to the correlation heat map and the fusion strategy decision maker in S4, including: Different fusion strategies are selected according to the high, medium and low correlation levels of the modal pairs in the correlation heat map. For highly correlated modal pairs, fusion is performed through shallow feature interaction to ensure that the features of the highly correlated modalities can jointly influence the model; for moderately correlated modal pairs, hybrid fusion is used to balance contributions; for low-correlation modal pairs, feature fusion is performed in depth to reduce interference between low-correlation modal pairs and avoid introducing noise.

6. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 1, characterized in that: The S5 step constructs an optimal fusion network architecture based on the high-dimensional features in the shared feature buffer pool and a specific fusion strategy, including: S51, initialize the number of fusion layers , defines the maximum number of fusion layers , initialize the candidate architecture pool to store the high-potential architectures screened out in each step; S52, for Layer fusion layer, using feature splicing to combine the first The output of the fusion layer is combined with the The features of the layer fusion layer are fused, and the features are allowed to jump between different layers. The fused features are nonlinearly transformed through the activation function to construct the candidate architecture; S53, for For all candidate architectures under the layer fusion layer, the sequential model based optimization SMBO algorithm is used, combined with the LSTM proxy model to predict the performance of the architecture; S54. Sort all candidate architectures according to performance, select multiple architectures with the highest performance as high-potential candidate architectures, and add them to the candidate architecture pool; S55. For high-potential candidate architectures, a weight inheritance mechanism is used to initialize parameters from the historical optimal layer for training; S56, at the maximum number of layers Under this condition, all high-potential candidate architectures in the candidate architecture pool are evaluated, and the architecture with the best performance is selected as the optimal fusion network architecture.

7. The multimodal automatic modeling and fusion method based on neural architecture search according to claim 1 is characterized in that: The step S6 automatically adjusts the output layer and loss function based on the optimal fusion network architecture and the task type through the task adaptation module to obtain a network that adapts to the task requirements, including: S61. Dynamically configure the output layer and loss function for the optimal fusion network architecture according to the task type; S62, dynamically adjusting the network structure and performing two-stage training on the configured optimal fusion network architecture; wherein, in the first stage, the parameters of the task adaptation module are fixed, and the feature representation of the configured optimal fusion network architecture is optimized; and in the second stage, the parameters of the optimal fusion network architecture are locked, and the task adaptation module is fine-tuned; S63. Determine whether the downstream task type has changed. If so, unload the output layer and loss function of the current task, load the configuration of the new task, and start hot start fine-tuning. If no change occurs, monitor the performance of the optimal fusion network architecture.

8. A multimodal automatic modeling and fusion device based on neural architecture search, wherein the multimodal automatic modeling and fusion device based on neural architecture search is used to implement the multimodal automatic modeling and fusion method based on neural architecture search according to any one of claims 1 to 7, characterized in that: The device comprises: Acquisition module, used to obtain multimodal data and task types; A neural architecture search module is used to automatically generate the corresponding optimal unit architecture for each modal data in the multimodal data based on the neural architecture search; wherein the optimal unit architecture is in the form of a directed acyclic graph; A single-modal feature extraction module is configured to use the optimal unit architecture as a feature extractor to perform feature extraction on each modal data, obtain high-dimensional features of each modal data, and store the high-dimensional features of each modal data in a shared feature buffer pool; A modality correlation calculation module is used to analyze the inter-modality dependency at the feature level based on the high-dimensional features of each modality data, generate a dynamically updated correlation heat map, and obtain a specific fusion strategy based on the correlation heat map and the fusion strategy decider; Dynamic fusion network module, used to build the optimal fusion network architecture based on the high-dimensional features in the shared feature cache pool and specific fusion strategies; The task adaptation module is used to automatically adjust the output layer and loss function according to the optimal fusion network architecture and task type, obtain a network that adapts to task requirements, and complete different downstream tasks.

9. A multimodal automatic modeling and fusion device, characterized in that: The multimodal automatic modeling and fusion device includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images