Automatic architecture design method and system for multi-modal fusion neural network
Through the automatic architecture design method of multimodal fusion neural network, the residual structure and Shapley value optimization are used to automatically generate multimodal fusion neural network structure, solving the problem of heterogeneous data integration and improving the flexibility of the model and resource utilization efficiency.
Patent Information
- Application Number
- CN202510362174.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-26
- Publication Date
- 2025-08-12
AI Technical Summary
The existing multimodal fusion neural network method cannot effectively integrate heterogeneous data, resulting in the model relying on some modal data and ignoring other modal information. Manual design limits model flexibility and causes waste of resources.
The automatic architecture design method of multimodal fusion neural network is adopted, and the residual structure is used to use multimodal fusion problems as architectural search problems. The neural network structure is automatically generated through the NAS search algorithm, and the network structure is optimized with Shapley value to generate the optimal multimodal fusion network backbone structure.
It has achieved efficient integration of information on various modalities, reduced the drawbacks of manual design, improved model stability and flexibility, and ensured the smooth completion of actual tasks.
Smart Images
Figure CN120471112A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of neural network architecture search technology, and in particular to a method and system for automatic architecture design of a multimodal fusion neural network. Background Art
[0002] Multimodal fusion technology can integrate information from multiple data sources such as images, text, and audio to provide more comprehensive data features, and thus more accurate predictions and decisions. Moreover, when data from a certain modality is missing or interfered with, multimodal technology can also use data from other modalities for predictions or decisions, which significantly improves the stability and reliability of the model.
[0003] There are some problems with the multimodal fusion methods in related technologies. For example, different types of data (such as images, text, and audio) have different feature spaces and data structures, and existing technologies cannot effectively fuse these heterogeneous data. In addition, in practical applications, there may be an imbalance in the quantity of data of different modalities, and the amount of data of some modalities is much larger than that of other modalities. This imbalance may cause the model to rely too much on data of certain modalities and ignore information of other modalities. In addition, the current multimodal fusion network architecture is mainly designed manually, which greatly limits the flexibility of the neural network model and causes a waste of resources such as manpower and time. Summary of the Invention
[0004] In order to solve the above technical problems, the purpose of the present invention is to provide an automatic architecture design method and system for a multimodal fusion neural network, which can use the residual structure to directly solve the multimodal fusion problem as an architecture search problem, and make full use of the various modal information of heterogeneous data to achieve data fusion.
[0005] The first technical solution adopted by the present invention is: a method for automatically designing the architecture of a multimodal fusion neural network, comprising the following steps:
[0006] Define the original operation pool for multimodal fusion neural network structure search and perform search processing to determine the multimodal fusion cell network structure;
[0007] Based on the multimodal fusion cell network structure, basic components for constructing multimodal fusion cell network structural units are obtained, and residual structures are inserted to obtain a multimodal fusion cell network unit library;
[0008] The topological complexity and accuracy of the multimodal fusion cell network unit library were evaluated, and the Shapley value of the residual structure in the multimodal fusion cell network structure was calculated to obtain the Shapley value of the residual structure;
[0009] The multimodal fusion cell network structure is optimized based on the Shapley value of the residual structure, and the optimal multimodal fusion cell network backbone structure is output;
[0010] The fusion of multiple sensor data is achieved based on the optimal multimodal fusion cell network backbone structure.
[0011] Furthermore, the step of obtaining basic components for constructing a multimodal fusion cell network structural unit based on the multimodal fusion cell network structure, inserting the residual structure, and obtaining a multimodal fusion cell network unit library specifically includes:
[0012] Based on the multimodal fusion cell network structure, basic components for constructing a multimodal fusion cell network structural unit are obtained;
[0013] Structural encoding of the basic components of the multimodal fusion cell network structural unit to generate a preliminary unit structure of the multimodal fusion network;
[0014] Based on the unit structure of the preliminary multimodal fusion network, the residual structure is inserted to obtain the multimodal fusion cell network unit library.
[0015] Furthermore, the basic components used to construct the multimodal fusion cell network structural unit specifically include a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, a 3x3 depth-separable convolutional layer, and a 5x5 depth-separable convolutional layer.
[0016] Furthermore, the step of evaluating the topological complexity and accuracy of the multimodal fusion cell network unit library, calculating the Shapley value of the residual structure in the multimodal fusion cell network structure, and obtaining the Shapley value of the residual structure specifically includes:
[0017] Evaluate the topological complexity and accuracy of the multimodal fusion cell network unit library to determine the topological complexity and accuracy of the unit structure of the multimodal fusion network;
[0018] According to the topological complexity value and accuracy value, the multimodal fusion cell network structure is iteratively selected and mutated, and the optimized multimodal fusion cell network structure is output;
[0019] Based on the optimized multimodal fusion cell network structure, the Shapley value of the residual structure in the multimodal fusion cell network structure is calculated to obtain the Shapley value of the residual structure.
[0020] Furthermore, the step of performing iterative selection and mutation operations on the multimodal fusion cell network structure according to the topological complexity value and the accuracy value, and outputting the optimized multimodal fusion cell network structure specifically includes:
[0021] According to the topological complexity value and accuracy value, the multimodal fusion cell network structure is selected and mutated;
[0022] Determine whether the current number of iterations t is an integer multiple of the time window Tk;
[0023] If t%Tk≠0, the selection and mutation operations on the multimodal fusion cell network structure are repeated until t%Tk==0, and the optimized multimodal fusion cell network structure is output.
[0024] Furthermore, the step of performing residual optimization processing on the multimodal fusion cell network structure according to the Shapley value of the residual structure and outputting the optimal multimodal fusion cell network backbone structure specifically includes:
[0025] The variation probability of the multimodal fusion cell network structure is adjusted according to the Shapley value of the residual structure to obtain the adjusted multimodal fusion cell network structure;
[0026] According to the adjusted multimodal fusion cell network structure, multiple cell libraries of different modalities are fused to obtain a new overall structure search space cell library;
[0027] Through the neural architecture search algorithm based on residual structure and Shapley value, the new overall structure search space cell library is searched to output the optimal multimodal fusion cell network backbone structure.
[0028] Furthermore, the step of searching the new overall structure search space cell library by using a neural architecture search algorithm based on residual structure and Shapley value to output the optimal multimodal fusion cell network backbone structure specifically includes:
[0029] Construct a preset Shapley threshold;
[0030] According to the preset Shapley threshold, the new overall structure search space cell library is screened to select the characteristic modes with Shapley values higher than the threshold in the new overall structure search space cell library;
[0031] The selected characteristic modalities are fused to obtain the optimal multimodal fusion cell network backbone structure.
[0032] Furthermore, the expression of the preset Shapley threshold is specifically as follows:
[0033]
[0034] In the above formula, φ irepresents the Shapley threshold, N represents N residual branches, S represents the substructure in the residual branch, f(S∪{i})-f(S) represents the marginal contribution of residual connection i to the performance improvement of the union of all structural combinations, Indicates that all possible structural combinations need to be summed. Indicates the weight of the structural combination.
[0035] The second technical solution adopted by the present invention is: an automatic architecture design system for a multimodal fusion neural network, comprising:
[0036] The first module is used to define the original operation pool of the multimodal fusion neural network structure search and perform search processing to determine the multimodal fusion cell network structure;
[0037] The second module is used to obtain the basic components used to construct the multimodal fusion cell network structure unit based on the multimodal fusion cell network structure, insert the residual structure, and obtain the multimodal fusion cell network unit library;
[0038] The third module is used to evaluate the topological complexity and accuracy of the multimodal fusion cell network unit library, calculate the Shapley value of the residual structure in the multimodal fusion cell network structure, and obtain the Shapley value of the residual structure;
[0039] The fourth module is used to perform residual optimization processing on the multimodal fusion cell network structure according to the Shapley value of the residual structure, and output the optimal multimodal fusion cell network backbone structure;
[0040] The fifth module is used to realize the fusion of multiple sensor data based on the optimal multimodal fusion cell network backbone structure.
[0041] The beneficial effects of the method and system of the present invention are as follows: the present invention determines the multimodal fusion cell network structure by defining the original operation pool of the multimodal fusion neural network structure search and performing search processing, and automatically generates the multimodal fusion neural network structure by using the NAS search algorithm, which can effectively reduce the disadvantages brought by traditional manual design, and then obtains the basic components for constructing the multimodal fusion cell network structure unit based on the multimodal fusion cell network structure, inserts the residual structure, and uses the residual structure to directly solve the multimodal fusion problem as an architecture search problem, and then evaluates the topological complexity and accuracy of the multimodal fusion cell network unit library, and performs Shapl on the residual structure in the multimodal fusion cell network structure. The ey value is calculated to obtain the Shapley value of the residual structure. The residual optimization processing of the multimodal fusion cell network structure is performed according to the Shapley value of the residual structure. The cell libraries of the above-mentioned multiple modalities are fused according to the preset Shapley value to generate a cell library of a new overall structure. The Shapley value is used to evaluate the contribution of each cell structure in multimodal fusion, thereby optimizing the fusion process. Finally, the NAS search algorithm is used to search the cell library of the new overall structure to determine the optimal neural network architecture. The optimal network architecture for a given data set is found through an efficient sequential model-based exploration method, which can fully integrate and utilize the information of each modality to ensure the smooth completion of the actual task. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a flowchart of the steps of an automatic architecture design method of a multimodal fusion neural network of the present invention;
[0043] Figure 2 This is a structural block diagram of an automatic architecture design system for a multimodal fusion neural network according to the present invention;
[0044] Figure 3 is a schematic diagram of the automatic architecture design of a multimodal fusion neural network provided by a specific embodiment of the present invention;
[0045] Figure 4 Schematic diagram of the design process of a Cell module of a single mode provided by a specific embodiment of the present invention;
[0046] Figure 5 This is a schematic diagram of the structure of the Cell structure encoding form of the multimodal fusion network provided by a specific embodiment of the present invention;
[0047] Figure 6 This is a schematic diagram of the principle of automatic design of a multimodal fusion network Cell structure provided by a specific embodiment of the present invention;
[0048] Figure 7This is a schematic diagram of the automatic design of the overall structure of a multimodal fusion neural network provided by a specific embodiment of the present invention;
[0049] Figure 8 It is a flowchart of a multimodal fusion neural network structure search provided by a specific embodiment of the present invention. DETAILED DESCRIPTION
[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.
[0051] First, it's important to note that unimodal technology relies solely on a single data source or sensor for information processing and decision-making. While effective in some situations, it also suffers from significant drawbacks, such as incomplete information, insufficient robustness, data sparsity, and uncertainty. These drawbacks are particularly pronounced when high accuracy, robustness, and complex environments are required.
[0052] To overcome these drawbacks, multimodal fusion technology has emerged as an effective solution. It integrates information from multiple data sources, such as images, text, and audio, providing more comprehensive data features and, in turn, more accurate predictions and decisions. Furthermore, when data from one modality is missing or interfered with, multimodal technology can leverage data from other modalities for predictions or decisions, significantly improving the stability and reliability of the model. This characteristic is particularly important in complex environments.
[0053] However, there are some problems with the multimodal fusion methods in the existing technology. For example, different types of data (such as images, text, and audio) have different feature spaces and data structures, and how to effectively fuse these heterogeneous data is a difficult problem. In addition, in practical applications, there may be an imbalance in the amount of data from different modalities, with the amount of data from some modalities being much larger than that from other modalities. This imbalance may cause the model to over-rely on data from certain modalities and ignore information from other modalities. In many application scenarios, multimodal fusion needs to meet the needs of real-time processing, which places higher demands on the performance of multimodal fusion algorithms. In addition, the current multimodal fusion network architecture is mainly designed manually, which greatly limits the flexibility of the neural network model and causes a waste of resources such as manpower and time. When processing multimodal tasks, the design of the multimodal fusion network architecture plays a very important role. The network architecture affects how to obtain effectively utilized information from multiple modalities.
[0054] Based on this, an embodiment of the present invention proposes an automatic architecture design method for a multimodal fusion neural network, which uses the NAS search algorithm to automatically generate a multimodal fusion neural network structure, which can effectively reduce the disadvantages brought by traditional manual design. In addition, an embodiment of the present invention proposes a new search space that covers a large number of possible fusion architectures, and finds the optimal network architecture for a given data set through an efficient sequential model-based exploration method, and for the first time uses a residual structure to directly solve the multimodal fusion problem as an architecture search problem. This allows the method of the present invention to fully integrate and utilize information from each modality to ensure the smooth completion of the actual task.
[0055] Reference Figure 1 The present invention provides an automatic architecture design method for a multimodal fusion neural network, the method comprising the following steps:
[0056] S100, defining an original operation pool for searching a multimodal fusion neural network structure and performing search processing to determine a multimodal fusion cell network structure;
[0057] In this embodiment, a neural architecture search (NAS) is performed on a plurality of different modalities to determine the backbone structure of the cell network of each modality.
[0058] Specifically, if Figure 4 As shown in Figure 1, the primitive operation pool for searching the multimodal fusion neural network structure is defined, which is the basic operation unit that can be selected. These operation units are different multimodal fusion neural network layers. Different multimodal fusion neural network structures can be constructed based on the combination of these operation units and their connection methods.
[0059] S200, based on the multimodal fusion cell network structure, obtaining basic components for constructing a multimodal fusion cell network structural unit, inserting the residual structure, and obtaining a multimodal fusion cell network unit library;
[0060] In this embodiment, based on the backbone structure of a cell network of multiple different modalities, a universal convolution operation is used as the basic component of automatic design to construct a residual structure, thereby realizing the residual connection on the cross-modal cell network backbone, and completing the automatic design of a cell library of multiple different modalities, that is, constructing a cell library containing multiple different modalities.
[0061] Specifically, if Figure 5 as well as Figure 6As shown in the figure, basic components for constructing a multimodal fusion network structural unit are obtained, which include a 1x1 convolution layer, a 3x3 convolution layer, a 5x5 convolution layer, a 3x3 depth-separable convolution layer, and a 5x5 depth-separable convolution layer; based on the basic components of the above multimodal fusion network structural units, structural encoding is performed to generate a preliminary unit structure of the multimodal fusion network, which adopts a backbone single-chain architecture including four layers of basic operations; based on the preliminary multimodal fusion network unit structure, a residual structure is inserted to complete the construction of the multimodal fusion network unit structure.
[0062] More specifically, to enhance design flexibility, the system employs a machine-generated strategy of creating multimodal cells with branches and various connectivity schemes. Before these cells are generated, the backbone structure of each cell is clearly defined to ensure versatility and scalability. The backbone structure of the multimodal cell utilizes the most basic and universal operations, forming a single chain consisting of four layers of fundamental operations as the cell's basic framework. When designing the structure of the multimodal cell, the machine can freely select and combine these fundamental operations, which facilitates the construction of more complex and efficient network architectures. This freedom enables the machine to automatically adjust and optimize the network structure based on the specific requirements of the data and task, resulting in a network structure that is more suitable for specific tasks. Furthermore, the introduction of skip operations allows information to be directly passed to deeper layers, further enhancing design flexibility. Furthermore, the residual structure can exist on the backbone chain structure in various combinations of input and output positions. This method allows each cell to have up to three branches, which can change the dimensionality of the input data.
[0063] S300, evaluating the topological complexity and accuracy of the multimodal fusion cell network unit library, calculating the Shapley value of the residual structure in the multimodal fusion cell network structure, and obtaining the Shapley value of the residual structure;
[0064] In this embodiment, the cell structure in the cell library is evaluated using topological complexity and accuracy, and selection and mutation operations are performed on the cell structure in the cell library to optimize the structure. It is determined whether the current number of iterations t is an integer multiple of the time window Tk, that is, whether t%Tk is equal to 0. If t%Tk≠0, the steps of evaluating the cell structure in the cell library using topological complexity and accuracy, and selecting and mutation operations are performed on the cell structure in the cell library to optimize the structure are repeated until t%Tk==0. If t%Tk==0, the Shapley value of the residual structure in the cell is calculated, and the mutation probability of the residual structure is adjusted according to the Shapley value. The number of repetitions of step S300 is the same as the number of modes.
[0065] Specifically, parameters such as the mutation probability, mutation parameters, accuracy, and topological structure complexity of the multimodal fusion network unit structure are obtained; the Shapley value of the residual structure in the multimodal fusion network unit structure is calculated to quantify the contribution of each residual structure; according to the calculated residual structure Shapley value, the mutation probability of the unit structure is adjusted to obtain the adjusted unit structure; the adjusted mutation parameters are used to perform individual mutation processing on the unit structure to generate the updated unit structure; and the accuracy and topological structure complexity of the updated multimodal fusion network unit structure are evaluated.
[0066] S400, performing residual optimization processing on the multimodal fusion cell network structure according to the Shapley value of the residual structure, and outputting the optimal multimodal fusion cell network backbone structure;
[0067] In this embodiment, a network structure design method based on the fusion of multiple different modalities based on Shapley values is used to fuse multiple different modal cell libraries to generate a new overall structure search space cell library, thereby achieving automatic search and design of the overall network structure of multiple different modalities. A neural architecture search algorithm based on residual structure and Shapley values is used to search the new overall structure search space cell library to determine the optimal neural network architecture.
[0068] Specifically, if Figure 7As shown, the residual optimization process is performed on the multimodal fusion network cell structure until the mutated and updated cell structure reaches the optimal level in terms of accuracy and topological complexity. The optimal multimodal fusion network cell structure is then output. During the search for the multimodal fusion network structure, a Shapley threshold is preset. Based on this threshold, characteristic modes in the multimodal cell structure with Shapley values above the threshold are screened. Subsequently, these screened modes are re-fused to construct a new overall search space for the multimodal fusion network. Finally, the optimal multimodal fusion network structure is searched using the Neural Network Architecture Search (NAS) method.
[0069] During the search for a multimodal fusion network structure, a Shapley threshold is preset. Based on this threshold, eigenmodes with Shapley values above the threshold are selected within the multimodal cell structure. These selected modes are then re-fused to construct a new overall search space for the multimodal fusion network. Finally, the optimal multimodal fusion network structure is found using a neural architecture search (NAS) method.
[0070] The Shapley threshold has good adaptive performance. The Shapley threshold is used to evaluate the contribution of each residual branch to the performance improvement of the cell structure. In the framework of cooperative game, it is assumed that N participants (i.e., residual branches) are interconnected, and a value function f maps each subset S of N to a real value f(S). f(S) represents the expected performance gain that the substructure composed of residual branches in S can obtain on the task. The formula f(S∪{i})-f(S) represents the marginal contribution of residual connection i to the performance improvement of the alliance of all structural combinations, and Indicates that all possible structural combinations need to be summed. The term represents the weight of the structural combination. The Shapley threshold of the residual branch i in the Cell structure can be calculated as:
[0071]
[0072] In the above formula, φ i represents the Shapley threshold, N represents N residual branches, S represents the substructure in the residual branch, f(S∪{i})-f(S) represents the marginal contribution of residual connection i to the performance improvement of the union of all structural combinations, Indicates that all possible structural combinations need to be summed. Indicates the weight of the structural combination.
[0073] S500, based on the optimal multimodal fusion cell network backbone structure, realizes the fusion of multiple sensor data.
[0074] Therefore, the multimodal fusion neural network in the embodiments of the present invention is a technology that can fuse data from multiple sensors (such as vision, lidar, radar, sonar, etc.) to provide more comprehensive and accurate environmental information. It has wide applications in fields such as autonomous driving, mobile robots, and drones, helping these systems better understand complex and dynamic operating environments, thereby making more accurate and safer decisions. In the field of autonomous driving, multimodal fusion networks can fuse data from multiple sensors such as cameras, lidar, and radar to provide vehicles with comprehensive, multi-level perception capabilities. This not only improves the recognition accuracy of static objects (such as traffic signs and road signs), but also effectively enhances the detection and tracking capabilities of dynamic objects (such as pedestrians and other vehicles). In addition, by fusing data from different sources, the robustness of the system can be enhanced. For example, in severe weather conditions, cameras may not work properly due to low visibility, but radar and lidar can still provide effective information. In the field of mobile robots, multimodal fusion networks can help robots better understand their operating environment, thereby achieving accurate navigation and positioning. For example, by fusing the precise distance information provided by lidar with the visual information captured by cameras, robots can more accurately construct maps of their surroundings, avoid obstacles, and find the most optimized path. Multimodal fusion also enhances robots' ability to operate in dynamic environments, such as safely moving around densely populated areas. In the field of drones, multimodal sensor fusion technology also plays a key role. It can help drones fly stably in a variety of complex environments, whether navigating obstacles between high-rise buildings in cities or through mountain forests. By fusing and analyzing data from cameras, radar, GPS, and other sources, drones can not only achieve high-precision positioning and navigation, but also complete more complex tasks such as target tracking, object recognition, and classification.
[0075] In summary, if Figure 3 as well as Figure 8As shown, an embodiment of the present invention first uses the NAS search algorithm to search for a variety of different modalities to determine the cell network (Cell Network) backbone structure of each modality, thereby constructing a cell library (Cell Library) containing a variety of different modalities. Then, the cell libraries of the above-mentioned multiple different modalities are fused according to the preset Shapley value to generate a cell library of a new overall structure. The Shapley value is used to evaluate the contribution of each cell structure in multimodal fusion, thereby optimizing the fusion process. Finally, the NAS search algorithm is used to search the cell library of the new overall structure to determine the optimal neural network architecture, and by comprehensively evaluating the accuracy and topological complexity of the network, the marginal contribution of the network performance and structure is quantified, thereby achieving an efficient network design with high precision, few parameters and simple structure. Specifically, the mutation probability parameter in the evolutionary algorithm is adjusted by calculating the Shapley value of the residual connection, thereby guiding the design direction of the automatic evolution of the structure.
[0076] Reference Figure 2 , an automatic architecture design system for multimodal fusion neural networks, including:
[0077] The first module 201 is used to define the original operation pool for multimodal fusion neural network structure search and perform search processing to determine the multimodal fusion cell network structure;
[0078] The second module 202 is used to obtain basic components for constructing multimodal fusion cell network structural units based on the multimodal fusion cell network structure, insert the residual structure, and obtain a multimodal fusion cell network unit library;
[0079] The third module 203 is used to evaluate the topological complexity and accuracy of the multimodal fusion cell network unit library, calculate the Shapley value of the residual structure in the multimodal fusion cell network structure, and obtain the Shapley value of the residual structure;
[0080] The fourth module 204 is used to perform residual optimization processing on the multimodal fusion cell network structure according to the Shapley value of the residual structure, and output the optimal multimodal fusion cell network backbone structure;
[0081] The fifth module 205 is used to realize the fusion of multiple sensor data based on the optimal multimodal fusion cell network backbone structure.
[0082] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0083] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A method for automatic architecture design of a multimodal fusion neural network, characterized in that: The following steps are involved: Define the original operation pool for multimodal fusion neural network structure search and perform search processing to determine the multimodal fusion cell network structure; Based on the multimodal fusion cell network structure, basic components for constructing multimodal fusion cell network structural units are obtained, and residual structures are inserted to obtain a multimodal fusion cell network unit library; The topological complexity and accuracy of the multimodal fusion cell network unit library were evaluated, and the Shapley value of the residual structure in the multimodal fusion cell network structure was calculated to obtain the Shapley value of the residual structure; The multimodal fusion cell network structure is optimized based on the Shapley value of the residual structure, and the optimal multimodal fusion cell network backbone structure is output; The fusion of multiple sensor data is achieved based on the optimal multimodal fusion cell network backbone structure.
2. The automatic architecture design method of a multimodal fusion neural network according to claim 1, characterized in that: The step of obtaining basic components for constructing a multimodal fusion cell network structural unit based on the multimodal fusion cell network structure, inserting the residual structure, and obtaining a multimodal fusion cell network unit library specifically includes: Based on the multimodal fusion cell network structure, basic components for constructing a multimodal fusion cell network structural unit are obtained; Structural encoding of the basic components of the multimodal fusion cell network structural unit to generate a preliminary unit structure of the multimodal fusion network; Based on the unit structure of the preliminary multimodal fusion network, the residual structure is inserted to obtain the multimodal fusion cell network unit library.
3. The automatic architecture design method of a multimodal fusion neural network according to claim 2, characterized in that: The basic components used to construct the multimodal fusion cell network structural unit specifically include a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, a 3x3 depth-separable convolutional layer, and a 5x5 depth-separable convolutional layer.
4. The automatic architecture design method of a multimodal fusion neural network according to claim 3, characterized in that: The step of evaluating the topological complexity and accuracy of the multimodal fusion cell network unit library, calculating the Shapley value of the residual structure in the multimodal fusion cell network structure, and obtaining the Shapley value of the residual structure specifically includes: Evaluate the topological complexity and accuracy of the multimodal fusion cell network unit library to determine the topological complexity and accuracy of the unit structure of the multimodal fusion network; According to the topological complexity value and accuracy value, the multimodal fusion cell network structure is iteratively selected and mutated, and the optimized multimodal fusion cell network structure is output; Based on the optimized multimodal fusion cell network structure, the Shapley value of the residual structure in the multimodal fusion cell network structure is calculated to obtain the Shapley value of the residual structure.
5. The automatic architecture design method of a multimodal fusion neural network according to claim 4, characterized in that: The step of performing iterative selection and mutation operations on the multimodal fusion cell network structure according to the topological complexity value and the accuracy value, and outputting the optimized multimodal fusion cell network structure specifically includes: According to the topological complexity value and accuracy value, the multimodal fusion cell network structure is selected and mutated; Determine whether the current number of iterations t is an integer multiple of the time window Tk; If t%Tk≠0, the selection and mutation operations on the multimodal fusion cell network structure are repeated until t%Tk==0, and the optimized multimodal fusion cell network structure is output.
6. The automatic architecture design method of a multimodal fusion neural network according to claim 5, characterized in that: The step of performing residual optimization processing on the multimodal fusion cell network structure according to the Shapley value of the residual structure and outputting the optimal multimodal fusion cell network backbone structure specifically includes: The variation probability of the multimodal fusion cell network structure is adjusted according to the Shapley value of the residual structure to obtain the adjusted multimodal fusion cell network structure; According to the adjusted multimodal fusion cell network structure, multiple cell libraries of different modalities are fused to obtain a new overall structure search space cell library; Through the neural architecture search algorithm based on residual structure and Shapley value, the new overall structure search space cell library is searched to output the optimal multimodal fusion cell network backbone structure.
7. The automatic architecture design method of a multimodal fusion neural network according to claim 6, characterized in that: The step of searching the new overall structure search space cell library by using the neural architecture search algorithm based on residual structure and Shapley value to output the optimal multimodal fusion cell network backbone structure specifically includes: Construct a preset Shapley threshold; According to the preset Shapley threshold, the new overall structure search space cell library is screened to select the characteristic modes with Shapley values higher than the threshold in the new overall structure search space cell library; The selected characteristic modalities are fused to obtain the optimal multimodal fusion cell network backbone structure.
8. The automatic architecture design method of a multimodal fusion neural network according to claim 7, characterized in that: The expression of the preset Shapley threshold is specifically as follows: In the above formula, φ i represents the Shapley threshold, N represents N residual branches, S represents the substructure in the residual branch, f(S∪{i})-f(S) represents the marginal contribution of residual connection i to the performance improvement of the union of all structural combinations, Indicates that all possible structural combinations need to be summed. Indicates the weight of the structural combination.
9. An automatic architecture design system for multimodal fusion neural networks, characterized in that: Includes the following modules: The first module is used to define the original operation pool of the multimodal fusion neural network structure search and perform search processing to determine the multimodal fusion cell network structure; The second module is used to obtain the basic components used to construct the multimodal fusion cell network structure unit based on the multimodal fusion cell network structure, insert the residual structure, and obtain the multimodal fusion cell network unit library; The third module is used to evaluate the topological complexity and accuracy of the multimodal fusion cell network unit library, calculate the Shapley value of the residual structure in the multimodal fusion cell network structure, and obtain the Shapley value of the residual structure; The fourth module is used to perform residual optimization processing on the multimodal fusion cell network structure according to the Shapley value of the residual structure, and output the optimal multimodal fusion cell network backbone structure; The fifth module is used to realize the fusion of multiple sensor data based on the optimal multimodal fusion cell network backbone structure.