Dynamic bypass-selected multi-modal network acceleration method and system

Abstract

A dynamic bypass selection multi-modal network acceleration method and system, after judging the original multi-modal data, the modal type is judged, and the bypass adaptation evaluation processing or the basic multi-modal model level reasoning is directly carried out, the bypass with the highest adaptation evaluation is selected as the result, and the subsequent execution scheme of the basic multi-modal model level reasoning is judged, the intermediate characteristics obtained by the basic multi-modal model level reasoning are aligned, and the bypass characteristics are generated through the bypass processing according to the selected bypass, and the specialized task results are obtained after the fusion of all bypass characteristics. The characteristics provided by different modal early levels can provide enough correct prediction requirements, and an improved dynamic bypass selection method is constructed, so that the overall multi-modal network calculation amount is reduced, the delay is reduced, and the partial precision is improved.

Description

Technical Field

[0001] This invention relates to a technology in the field of neural network applications, specifically a method and system for accelerating multimodal networks with dynamic bypass selection. Background Technology

[0002] Multimodal neural networks (NNs) integrate information from different modalities (such as text, images, and audio) to achieve more comprehensive learning and analysis. Compared to traditional unimodal neural networks, NNs can study a single object from multiple perspectives, leading to more accurate analysis and prediction. Existing work has achieved higher model performance and stability through multimodal network integration. However, this approach requires full inference across all modalities, resulting in greater overhead, higher computational consumption, and memory usage, which can lead to unnecessary resource waste for some simple tasks. Summary of the Invention

[0003] This invention addresses the problems of high cost and increased overall application latency caused by multiple repetitive encoders and additional fusion operations in existing technologies, as well as the significant decrease in inference accuracy due to differences in the expressive capabilities of different modalities or overfitting in existing multimodal technologies. It proposes a dynamic bypass selection method and system for accelerating multimodal networks. Based on the characteristic that the features provided by the early layers of different modalities can provide sufficient accuracy for prediction, an improved dynamic bypass selection method is constructed to significantly reduce the overall computational load of the multimodal network, reduce latency, and improve some accuracy.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a method for accelerating multimodal networks with dynamic bypass selection. After determining the modality type of the original multimodal data, bypass fitness evaluation is performed or basic multimodal model hierarchical inference is performed directly. The bypass with the highest fitness evaluation is selected as the result to determine the subsequent execution plan for basic multimodal model hierarchical inference. The intermediate features obtained from basic multimodal model hierarchical inference are aligned and bypass features are generated based on the selected bypass after bypass processing. After fusing all bypass features, the specialized task result is obtained through inference.

[0006] The raw multimodal data, i.e. the raw input of the multimodal task, includes, but is not limited to, image, text and voice information, and its data sources include, but are not limited to, cameras, sensors, user input text and microphones.

[0007] Feature alignment refers to transforming features that are determined to be image types by a neural network into a form with the same dimension as the final output features of that modality.

[0008] The bypass fit evaluation process refers to: using a simplified model to extract features from the original multimodal data to obtain simplified features for each modality; using the simplified features of each modality to obtain multimodal features through feature fusion; and using these multimodal features to evaluate the fit of each bypass.

[0009] The simplified model refers to a low-cost neural network model constructed for different modal data, used to extract features with certain expressive power from the original multimodal data. This includes, but is not limited to, using the ResNet model for feature extraction for image modal data and using the Attention model for feature extraction for text and speech modal data.

[0010] The aforementioned basic multimodal model refers to a neural network model used to extract highly expressive features of each modality from the original multimodal data. This includes, but is not limited to, the ResNet model for extracting features from image modalities, the BERT model for extracting features from text modalities, and the Liborosa model for extracting features from speech modalities. Specifically, it includes multiple sequentially connected neural network layers, where the input of each neural network layer is the output of the previous neural network layer, and the input of the first neural network layer is the original multimodal data. Hierarchical inference is achieved through layer-by-layer feature extraction.

[0011] The intermediate features mentioned refer to the features output by one of the neural network layers during the hierarchical inference process of the basic multimodal model.

[0012] During the hierarchical inference process of the basic multimodal model, after each neural network layer, it is determined whether the current layer has reached the layer corresponding to the bypass selection result. If it has reached the layer, the intermediate features output by that layer are output to the corresponding bypass processing; otherwise, the next level of inference of the basic multimodal model is performed. After the inference is completed, the comparison step is repeated.

[0013] The bypass processing refers to obtaining intermediate features in the hierarchical inference process of the basic multimodal model, performing feature alignment processing, and then using a low-computing-power neural network to obtain the final output features of the basic multimodal model.

[0014] The feature alignment process refers to the following: for a modality whose modality type is determined to be an image, a low-computing-power neural network is used to transform the input features into a form with the same dimension as the final output features of that modality.

[0015] The feature fusion refers to merging the final output features of each modality through a neural network to obtain a multimodal feature that covers all modal information.

[0016] Technical effect

[0017] This invention addresses feature extraction from various modal data through a basic model module; by introducing a bypass module for dynamic adaptation, it can match and fuse networks of different dimensions and types to generate dynamic bypasses, enabling the system to be widely adapted to existing network models; by introducing a bypass selection module, it filters and compares to obtain the optimal bypass inference combination in terms of accuracy and time consumption with minimal time and space overhead. Compared with existing technologies, this invention significantly improves the resource utilization and time efficiency of multimodal network systems while ensuring the performance of multimodal networks. Attached Figure Description

[0018] Figure 1 This is a flowchart of the present invention;

[0019] Figure 2 This is a schematic diagram of the system structure of the present invention;

[0020] Figure 3 This is a schematic diagram illustrating the execution process of an example.

[0021] Figure 4 A comparison chart showing the execution latency and total latency savings of each module of this invention under different datasets. Detailed Implementation

[0022] like Figure 2 As shown in this embodiment, a multimodal network acceleration system with dynamic bypass selection is included. It comprises an integrated inference module containing a distributor, a controller, a fusion module, and an inference module, and bypass modules connected to it respectively. Specifically: the integrated inference module sends raw multimodal data to the bypass selection module and the basic model module via the distributor; the controller uses the bypass selection results obtained from the bypass selection module to perform hierarchical judgments on the inference process of the basic model module and controls its operation; the fusion module fuses the modal features output by the bypass modules to obtain multimodal features with high expressive power and covering all modal information; and the inference module processes the data obtained from the fusion module. The multimodal features are used for inference to obtain specialized task results; the bypass selection module extracts shallow features of each modality data and evaluates the fit of the bypass network corresponding to each modality in the bypass module, and outputs the corresponding optimal bypass selection result to the integrated inference module; the basic model module performs hierarchical inference of the corresponding modality, and after each hierarchical inference, the feature data obtained is used to continue hierarchical inference of the next layer or output to the bypass module according to the result of the controller's hierarchical judgment; the bypass module uses the corresponding bypass network to extract features and generate secondary features of each modality, and then outputs them to the fusion unit of the integrated inference module. The inference unit based on different task requirements performs corresponding task outputs according to the fused features.

[0023] The aforementioned corresponding modality initial hierarchical inference refers to: using the basic feature extraction structure of a basic residual network for image data, and selecting an attention mechanism as the basic feature extraction structure for text data, in order to meet the model feature extraction requirements of the task.

[0024] The basic model module, after feature extraction at each level, outputs the sequence number of the currently completed level to the controller. It compares the optimal bypass selection output by the bypass selection module with the currently completed level. If they are equal, the features obtained from the currently completed level are output to the bypass module for distribution; otherwise, inference at the next level is performed. This basic model module specifically includes: multiple single-modality multi-level neural networks. For any modality of multi-level neural network, the feature extraction process for each multi-level neural network is: x i =F i (x i-1 ) = F i (F i-1 (…F1(x0))), where: x i F represents the features obtained after feature extraction from the first i layers of this single-modality multi-level neural network. i (·) represents the i-th layer of the network, and x0 represents the original input of the mode.

[0025] The bypass selection module includes a feature extraction unit, a feature fusion unit, and a bypass fitness evaluation unit. The feature extraction unit performs fast feature extraction for each modality based on the original multimodal data information to obtain shallow features for each modality. The feature fusion unit performs fusion processing based on the shallow features of each modality to obtain multimodal features. The bypass fitness evaluation unit evaluates the fitness of each bypass network based on the multimodal features to obtain the optimal bypass combination that can obtain accurate results with minimal inference latency.

[0026] The bypass module includes multiple bypass networks, wherein each bypass network corresponds to each layer of each single-modality multi-level neural network in the basic model module, and each bypass network extracts highly expressive features of the corresponding modality that can be used for ensemble reasoning based on the intermediate features of the basic model module.

[0027] The bypass network includes a feature alignment unit and a feature extraction unit. The feature alignment unit performs feature alignment processing on the intermediate features based on modality type judgment information, intermediate feature information obtained from the basic model module, and the deepest feature dimension information to obtain features with the same dimension as the deepest feature. The feature extraction unit performs secondary feature extraction processing based on the features obtained by the dimensionality reduction unit to obtain modality features that can be used for fusion inference to obtain accurate results.

[0028] In this embodiment, the bypass network is implemented using a low-computing-power version of the encoder.

[0029] The described integrated inference module includes: a distributor, a controller, a fuser, and an inferencer, where: the distributor distributes the original multimodal data of the task input to the bypass selection module and the basic model module; the controller receives the information of the optimal bypass network from the bypass selection module to control the operation of the basic model modality; the fuser receives the highly expressive features of each modality from the bypass module and fuses them to obtain multimodal features; the inferencer uses the multimodal features to infer and obtain the specialized task result.

[0030] The control of the operation of the basic model modality means that: for any modality, when the optimal bypass selection obtained by the bypass selection module is D, record that the neural network of this modality in the basic model modality is currently executed to the i-th layer to obtain the feature x i , when i < D, continue to execute the network in the basic model modality, that is, x i+1 = F i+1 (x i ), otherwise (i = D), execute the corresponding bypass network in the bypass module, y = P i (x i ), to obtain the output result of the bypass module of this modality.

[0031] As Figure 1 shown, in this embodiment, after performing modality judgment on the type of the original multimodal data, the original data is used by a simplified model for shallow feature extraction, the extracted shallow features are used for bypass adaptability evaluation, and the corresponding bypass selection result is output according to the maximum value. Then, the collected original data is used by the basic model for hierarchical inference, and it is compared with the bypass selection result output by the bypass selection module and the current hierarchical result to determine whether to output; finally, all the multimodal intermediate features output by the bypass module are processed by the bypass, the generated bypass features are used for feature fusion, and the fusion features are used for outputting the specialized task result for the task. The results are shown in Table 1, Figure 4 as

[0032] Table 1

[0033]

[0034] The three technical solutions in Table 1 are respectively represented as: (a) LateFusion is a traditional late fusion model; (b) MMExit is a multimodal network acceleration technology based on early exit; (c) is the multimodal network acceleration technology and system based on dynamic bypass selection in this embodiment. The three performance indicators used in the table are respectively: (a) Acc is the accuracy rate of the embodiment result, corresponding to the precision of the embodiment result; (b) F1 is the F1 score of the embodiment result, corresponding to the precision of the embodiment result; (c) Time is the execution time of the embodiment, corresponding to the latency of the embodiment result.

[0035] ① Table 1 compares the latency and accuracy of this embodiment and existing multimodal networks on four multimodal datasets. The data in the table shows that this invention achieves the best accuracy and latency across all multimodal datasets.

[0036] ② Figure 4 The latency of each module in this embodiment is compared with that of the traditional late fusion model. As can be seen from the results in the figure, although the bypass module and bypass selection module themselves introduce additional latency, they also reduce the latency of the original base model module and integrated inference module, resulting in a latency reduction of 16% to 42% compared to the late fusion model.

[0037] Compared with existing technologies, this method introduces a bypass selection module, which eliminates the need for full inference in most cases, reducing resource consumption and saving time. At the same time, by introducing a bypass module, it adapts to the output of each level, has high adaptability, reasonably matches task requirements, and reduces the overall task accuracy loss. The introduction of an integrated inference module coordinates the allocation of data and the logical link between the two modules mentioned above, unifying the logical sequence of system execution.

[0038] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for accelerating multimodal networks with dynamic incremental bypass selection, characterized in that, After determining the modality type of the original multimodal data, bypass fitness evaluation is performed or basic multimodal model hierarchical inference is performed directly. The bypass with the highest fitness evaluation is selected to determine the subsequent execution plan of basic multimodal model hierarchical inference. The intermediate features obtained from basic multimodal model hierarchical inference are aligned and processed according to the selected bypass to generate bypass features. After fusing all bypass features, the specialized task result is obtained through inference. The bypass fit evaluation process refers to: using a simplified model to extract features from the original multimodal data to obtain simplified features for each modality; using the simplified features of each modality to obtain multimodal features through feature fusion; and using these multimodal features to evaluate the fit of each bypass, i.e., obtaining accurate results with minimal inference latency. The simplified model refers to a low-cost neural network model constructed for different modal data, used to extract features with certain expressive power from the original multimodal data, including using the ResNet model for feature extraction for image modal data and using the Attention model for feature extraction for text and speech modal data.

2. The multimodal network acceleration method for dynamic incremental bypass selection according to claim 1, characterized in that, The basic multimodal model consists of multiple sequentially connected neural network layers. The input of each neural network layer is the output of the previous neural network layer. The input of the first neural network layer is the original multimodal data. Hierarchical inference is achieved through layer-by-layer feature extraction.

3. The multimodal network acceleration method for dynamic incremental bypass selection according to claim 1, characterized in that, During the hierarchical inference process of the basic multimodal model, after each neural network layer, it is determined whether the current layer has reached the layer corresponding to the bypass selection result. If it has reached the layer, the intermediate features output by that layer are output to the corresponding bypass processing; otherwise, the next level of inference of the basic multimodal model is performed. After the inference is completed, the comparison step is repeated.

4. A multimodal network acceleration system for implementing dynamic bypass selection of the method described in any one of claims 1-3, characterized in that, include: An integrated inference module comprising a distributor, controller, fusion unit, and inference unit, and connected to it by a bypass selection module, a base model module, and bypass modules, wherein: the integrated inference module sends raw multimodal data to the bypass selection module and the base model module respectively through the distributor; the controller uses the bypass selection results obtained from the bypass selection module to perform hierarchical judgment on the inference process of the base model module and controls the operation of the base model module; the fusion unit fuses the modal features output by the bypass modules to obtain multimodal features with high expressive power and covering all modal information; the inference unit infers from the multimodal features obtained by the fusion unit to obtain specialized tasks. The bypass selection module extracts shallow features from each modality's data and evaluates the fit of the bypass network corresponding to each modality in the bypass module, outputting the corresponding optimal bypass selection result to the integrated inference module; the basic model module performs hierarchical inference for the corresponding modality, and after each hierarchical inference, it continues to perform hierarchical inference for the next layer or outputs the obtained feature data to the bypass module based on the result of the controller's hierarchical judgment; the bypass module uses the corresponding bypass network to extract features and generate secondary features for each modality, then outputs them to the fusion unit of the integrated inference module, and the inference unit based on different task requirements performs corresponding task outputs based on the fused features; Corresponding modality initial hierarchical inference refers to: using the basic feature extraction structure of a basic residual network for image data, and selecting an attention mechanism as the basic feature extraction structure for text data, in order to meet the model feature extraction requirements that meet the task requirements.

5. The multimodal network acceleration system according to claim 4, characterized in that, The basic model module, after feature extraction at each level, outputs the sequence number of the currently completed level to the controller. It compares the optimal bypass selection output by the bypass selection module with the currently completed level. If they are equal, the features obtained from the currently completed level are output to the bypass module for distribution; otherwise, inference at the next level is performed. This basic model module specifically includes: multiple single-modality multi-level neural networks. For any modality of multi-level neural network, the feature extraction process for each multi-level neural network is as follows: ,in: This single-modal multi-level neural network has undergone the previous... The features obtained after feature extraction of the layer, For the network's first Layered network, This is the original input for this mode.

6. The multimodal network acceleration system according to claim 4, characterized in that, The bypass selection module includes a feature extraction unit, a feature fusion unit, and a bypass fitness evaluation unit. The feature extraction unit performs fast feature extraction for each modality based on the original multimodal data information to obtain shallow features for each modality. The feature fusion unit performs fusion processing based on the shallow features of each modality to obtain multimodal features. The bypass fitness evaluation unit evaluates the fitness of each bypass network based on the multimodal features to obtain the optimal bypass combination that can obtain accurate results with minimal inference latency.

7. The multimodal network acceleration system according to claim 4, characterized in that, The bypass module includes: multiple bypass networks, wherein: each bypass network corresponds to each layer of each single-modality multi-level neural network in the basic model module, and each bypass network extracts highly expressive features of the corresponding modality that can be used for integrated reasoning based on the intermediate features of the basic model module; The bypass network includes a feature alignment unit and a feature extraction unit. The feature alignment unit performs feature alignment processing on the intermediate features based on modality type judgment information, intermediate feature information obtained from the basic model module, and the deepest feature dimension information to obtain features with the same dimension as the deepest feature. The feature extraction unit performs secondary feature extraction processing based on the features obtained by the dimensionality reduction unit to obtain modality features that can be used for fusion inference to obtain accurate results.

8. The multimodal network acceleration system according to claim 4, characterized in that, The integrated inference module includes a distributor, a controller, a fusion unit, and an inference unit. The distributor distributes the raw multimodal data of the task input to the bypass selection module and the base model module. The controller receives information about the optimal bypass network from the bypass selection module to control the operation of the base model modal. The fusion unit receives highly expressive features of each modality from the bypass module and fuses them to obtain multimodal features. The inference unit uses the multimodal features to infer the specialized task results. The operation of the control basic model modes mentioned above refers to: for any mode, when the optimal bypass selection obtained by the bypass selection module is... Records the current execution level of the neural network in the base model mode up to the specified number of iterations. Layers obtain features ,when Then continue executing the network in the basic model modality, i.e. ,when The corresponding bypass network in the bypass module is executed at that time. This yields the output of the bypass module for that mode.

Images