Multi-modal end-edge collaboration system and method based on hybrid expert model

Abstract

The application discloses a multi-modal end-edge collaborative system and method based on a mixed expert model, which comprises a multi-modal data collector configured to collect original multi-modal data; a shared feature extractor configured to perform feature extraction on the original multi-modal data to obtain intermediate features; a feature compression encoder configured to compress the intermediate features to obtain final compressed features and send the final compressed features to an edge server; a feature recovery decoder configured to perform high-fidelity recovery on the final compressed features to obtain a reconstructed feature map; a grid-level sparse MoE module configured to output an inference result based on the reconstructed feature map and a preset grid-level sparse mixed expert model; and a feature importance analysis module configured to construct a grid-level importance map, solve based on the grid-level importance map, and generate an adaptive transmission decision. The embodiment of the application provides a feasible and systematic solution for multi-modal large model collaborative deployment in a resource-limited network.

Description

Technical Field

[0001] This application relates to the field of multimodal intelligent sensing technology, and in particular to a multimodal edge-end collaborative system and method based on a hybrid expert model. Background Technology

[0002] With the rapid development of 6G mobile communication networks, the Internet of Things (IoT), and artificial intelligence (AI) technologies, the number of terminal devices at the network edge is constantly increasing, and the data types are becoming increasingly diverse. Intelligent applications are placing higher demands on low-latency response, privacy protection, and localized processing capabilities. Against this backdrop, more and more intelligent applications need to perform real-time perception, understanding, and decision-making on terminal devices or edge nodes close to the terminal. In practical applications, terminal devices typically acquire environmental information through cameras, microphones, radar, positioning modules, and various sensors, thereby forming multimodal data such as images, videos, audio, location, and sensor signals. This data is then combined with text commands, task descriptions, and other information to constitute multimodal data. For example, in scenarios such as intelligent monitoring, industrial vision, autonomous driving, and robot control, systems often need to jointly analyze the aforementioned multimodal data to achieve more accurate scene perception and task decision-making.

[0003] Faced with the aforementioned multimodal data, compared to intelligent models that rely solely on single-modal data, multimodal large language models can integrate complementary information from different modalities, thereby achieving stronger environmental perception, semantic understanding, and reasoning capabilities. Since different modalities of data exhibit significant differences in data format, semantic features, and task contributions, models need the ability to perform differentiated modeling and collaborative processing for different types of information. Hybrid expert models typically consist of multiple expert subnetworks, capable of dynamically allocating data with different modalities, semantics, or task characteristics to the corresponding expert subnetworks through gating mechanisms, making them suitable for multimodal data processing scenarios. For example, for data containing images and text, the model can dynamically invoke visual experts, language experts, or cross-modal fusion experts for processing based on image regions, text semantics, and the relationships between the two. Based on the aforementioned dynamic routing and expert collaboration mechanisms, the hybrid expert model architecture can improve the flexibility and reasoning efficiency of multimodal data processing, gradually becoming one of the important technical paths for constructing multimodal large models, and demonstrating strong application potential in tasks such as complex scene understanding, object recognition, and cross-modal reasoning.

[0004] However, existing multimodal large models based on hybrid expert model architectures are primarily designed for server or high-performance computing platforms, which typically possess strong computing capabilities. In contrast, edge devices such as mobile terminals, IoT sensors, and wearable devices are usually limited by their computing power, making it difficult to directly deploy multimodal large models with large parameter scales and high computational complexity. Therefore, how to extend the inference capabilities of multimodal large models to resource-constrained edge scenarios has become a crucial problem that urgently needs to be solved in the fields of edge intelligence and edge computing.

[0005] To alleviate the problem of insufficient computing resources on edge devices, various edge-edge collaborative inference schemes have been proposed in existing technologies. Some of these schemes deploy the gating network on edge servers and distribute the expert network across different terminal devices, thus alleviating computational pressure to some extent by splitting the computational tasks. However, these schemes require all raw multimodal data to be uploaded to the edge server for routing decisions, resulting in significant uplink communication overhead. For example, in video surveillance scenarios, continuously uploading high-definition video streams will quickly exhaust the limited bandwidth of the wireless network.

[0006] Furthermore, existing edge-end collaboration solutions lack fine-grained selection mechanisms for the transmitted content of multimodal data, often failing to adequately differentiate the importance of different modalities, samples, and feature regions to the final inference result. In practical multimodal tasks, the contributions of different modalities are not equal; even within the same modality, the contributions of different samples and different local feature regions to the task result may vary significantly. Transmitting all data or a large number of intermediate features indiscriminately will waste communication resources; and under bandwidth-constrained conditions, this transmission method may also prevent the priority transmission of effective information, thereby affecting the final inference performance.

[0007] In summary, the existing technologies have the following shortcomings: First, large multimodal models based on hybrid expert models cannot be directly deployed on resource-constrained edge devices due to their huge computational requirements; second, although existing edge-edge collaboration solutions attempt to split the model to the edge and the end, both require uploading raw data or a large number of intermediate features, resulting in excessive communication overhead and making them unsuitable for bandwidth-constrained real-world network environments; third, under limited bandwidth conditions, there is a lack of a unified optimization mechanism for multi-level decisions such as modality selection, sample screening, and feature transmission, making it impossible to optimize system performance under communication resource constraints. Summary of the Invention

[0008] This application provides a multimodal edge-cooperative system based on a hybrid expert model. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0009] In a first aspect, embodiments of this application provide a multimodal edge-to-edge collaborative system based on a hybrid expert model, the system comprising: At least one terminal node and an edge server are configured; the terminal node is equipped with a multimodal data acquisition unit, a shared feature extractor, and a feature compression encoder; the edge server is equipped with a feature recovery decoder, a grid-level sparse MoE module, and a feature importance analysis module; among which... A multimodal data acquisition unit is used to collect raw multimodal data; a shared feature extractor is used to extract features from the raw multimodal data to obtain intermediate features. The feature compression encoder is used to compress intermediate features to obtain the final compressed features, which are then sent to the edge server. The feature recovery decoder is used to perform high-fidelity recovery of the final compressed features to obtain the reconstructed feature map; The grid-level sparse MoE module is used to output inference results based on reconstructed feature maps and a preset grid-level sparse hybrid expert model. The feature importance analysis module is used to construct a grid-level importance graph, solve for the grid-level importance graph, and generate adaptive transmission decisions to guide terminal nodes in uploading features.

[0010] Secondly, a multimodal edge collaboration method based on a hybrid expert model is proposed, the method comprising: A multimodal data acquisition unit collects raw multimodal data; a shared feature extractor is used to extract features from the raw multimodal data to obtain intermediate features. The feature compression encoder compresses the intermediate features to obtain the final compressed features, which are then sent to the edge server. The feature restoration decoder performs high-fidelity restoration of the final compressed features to obtain the reconstructed feature map; The grid-level sparse MoE module outputs inference results based on reconstructed feature maps and a pre-defined grid-level sparse hybrid expert model. The feature importance analysis module constructs a grid-level importance graph, solves the problem based on the grid-level importance graph, and generates adaptive transmission decisions to guide terminal nodes in uploading features.

[0011] In this embodiment, by deploying a lightweight shared feature extraction module and a feature compression encoding module on the edge side, and deploying a feature reconstruction module and a grid-level sparse hybrid expert model on the edge server, the multimodal large model inference capability is efficiently extended to resource-constrained terminal scenarios, significantly reducing uplink communication overhead. Simultaneously, by introducing a feature importance analysis mechanism based on Shapley values, modality selection, sample screening, and feature transmission granularity are jointly modeled as a knapsack problem with grouping constraints for optimization. This enables dynamic selection of the most valuable features for transmission under bandwidth-constrained conditions, maintaining high inference performance while reducing communication overhead. This achieves an effective trade-off between communication resources and inference accuracy, providing a feasible systematic solution for the collaborative deployment of multimodal large and small models in resource-constrained networks.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0014] Figure 1 This is a schematic diagram of the system structure of a multimodal edge-to-edge collaborative system based on a hybrid expert model, provided in an embodiment of this application. Figure 2 This is a flowchart illustrating a nonmaximum suppression method for joint selection of modality, sample, and feature provided in this application. Figure 3 This is a flowchart illustrating a multimodal end-edge collaboration method based on a hybrid expert model provided in an embodiment of this application. Detailed Implementation

[0015] The following description and accompanying drawings fully illustrate specific embodiments of this application to enable those skilled in the art to practice them.

[0016] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0017] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with some aspects of this application as detailed in the appended claims.

[0018] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0019] In this embodiment, by deploying a lightweight shared feature extraction module and a feature compression encoding module on the edge side, and deploying a feature reconstruction module and a grid-level sparse hybrid expert model on the edge server, the multimodal large model inference capability is efficiently extended to resource-constrained terminal scenarios, significantly reducing uplink communication overhead. Simultaneously, by introducing a feature importance analysis mechanism based on Shapley values, modality selection, sample screening, and feature transmission granularity are jointly modeled as a knapsack problem with grouping constraints for optimization. This enables dynamic selection of the most valuable features for transmission under bandwidth-constrained conditions, maintaining high inference performance while reducing communication overhead. This achieves an effective trade-off between communication resources and inference accuracy, providing a feasible systematic solution for the collaborative deployment of multimodal large and small models in resource-constrained networks.

[0020] Please see Figure 1 , Figure 1 This is a schematic diagram of the system architecture of a multimodal edge-end collaborative system based on a hybrid expert model, provided in an embodiment of this application. The system includes: at least one terminal node and an edge server; the terminal node is equipped with a multimodal data collector, a shared feature extractor, and a feature compression encoder; the edge server is equipped with a feature recovery decoder, a grid-level sparse MoE module, and a feature importance analysis module; wherein, the multimodal data collector is used to collect raw multimodal data; the shared feature extractor is used to extract features from the raw multimodal data to obtain intermediate features; the feature compression encoder is used to compress the intermediate features to obtain the final compressed features, which are then sent to the edge server; the feature recovery decoder is used to perform high-fidelity recovery of the final compressed features to obtain a reconstructed feature map; the grid-level sparse MoE module is used to output inference results based on the reconstructed feature map and a preset grid-level sparse hybrid expert model; the feature importance analysis module is used to construct a grid-level importance map, solve the grid-level importance map, and generate adaptive transmission decisions to guide the terminal node in uploading features.

[0021] In this embodiment, taking a multimodal target detection task as an example, to facilitate large model analysis on the edge server, the edge node first processes the original multimodal data using a lightweight shared feature extraction module to extract intermediate features. Subsequently, the intermediate features are dimensionality-reduced using a feature compression encoding module to reduce transmission overhead. The compressed features are then uploaded to the edge server via a wireless channel. Upon receiving the compressed features, the edge server performs high-fidelity reconstruction using a feature reconstruction module and inputs the reconstructed features into a grid-level sparse MoE module. This module divides the features into fine-grained grid cells, and a dynamic gating network routes each grid cell to the most relevant Top-k expert subnetwork for inference, thereby achieving an effective balance between computational efficiency and model capacity.

[0022] It should be noted that, considering a containing A set of end nodes = Each node Equipped with multimodal sensors to generate multimodal data Each endpoint deploys a lightweight shared feature extraction backbone network to extract high-dimensional feature maps from the raw data. This feature map can be considered as A set of local feature vectors, each vector Semantic information corresponding to a specific spatial region in the original input. Feature map. After compression, the data is transmitted to the edge server, where it is reconstructed into an approximate feature map. And used for the final multimodal task.

[0023] In some embodiments of this application, the shared feature extractor is implemented based on a ConvNeXt-T backbone network, which includes a first processing stage, a second processing stage, a third processing stage, and a fourth processing stage arranged sequentially. The first and second processing stages are deployed on terminal nodes; the third and fourth processing stages are deployed on edge servers and integrated with a grid-level sparse MoE module. The first and second processing stages are used to perform shallow feature extraction on the original multimodal data and output intermediate features. The third and fourth processing stages are used to receive the reconstructed feature map output by the feature recovery decoder and perform sparse expert inference based on the reconstructed feature map and the grid-level sparse MoE module.

[0024] In one possible implementation, to achieve efficient multimodal feature extraction at edge nodes while avoiding parameter redundancy caused by deploying networks independently for each modality, this application employs a shared feature extractor. This shared feature extractor is based on the ConvNeXt-T backbone network, and its structure adopts a four-stage hierarchical design: each stage consists of multiple ConvNeXt blocks, with spatial resolution gradually reduced and channel number increased through downsampling layers. The core component is a depthwise separable convolution, which expands the receptive field while maintaining high computational efficiency. Considering the limitations of edge computing resources, this application proposes an edge-edge collaborative deployment strategy: the first two stages of ConvNeXt-T are deployed on edge nodes, and the last two stages are deployed on edge servers. The advantages of this strategy are: the feature maps of the first two stages have a larger spatial size and fewer channels, with approximately 0.32M parameters, making them suitable for edge operation. Simultaneously, local processing avoids directly transmitting the bandwidth-intensive original image, requiring only the transmission of intermediate features, which helps reduce communication overhead. The last two stages on the edge side can be flexibly adjusted according to different application requirements without modifying the edge model, ensuring the scalability of the system.

[0025] The feature compression encoder adopts a residual dual-branch structure, which includes a residual branch and a main branch.

[0026] In some embodiments of this application, the specific process of compressing intermediate features to obtain final compressed features and sending them to the edge server includes: the residual branch compresses the intermediate features to obtain a first compressed feature; the main branch sequentially integrates spatial attention and channel attention mechanisms for weighted processing of the intermediate features to obtain a second compressed feature; the first compressed feature and the second compressed feature are concatenated to obtain an initial compressed feature. For each element in the initial compressed feature, after pruning and quantization, the quantized feature is output; Gaussian noise is added to the quantized feature to obtain a noise-enhanced final compressed feature; the noise-enhanced final compressed feature is sent to the edge server via a wireless link.

[0027] Specifically, the process of integrating spatial attention and channel attention mechanisms sequentially to weight the intermediate features and obtain the second compressed feature includes: reducing the dimensionality of the intermediate features by using convolutional layers to obtain dimensionality-reduced features; applying spatial attention and channel attention mechanisms to the dimensionality-reduced features to obtain features with dual attention weighting; calculating the mean and maximum values ​​of the feature map in the channel dimension for the dimensionality-reduced features using the spatial attention mechanism, concatenating the two and generating a spatial attention map through a 7×7 convolution; extracting channel descriptors for the dimensionality-reduced features using global average pooling and global max pooling, and then generating channel attention weights through a shared MLP; and sequentially passing the features with dual attention weighting through a 3×3 depthwise separable convolution, batch normalization, ReLU activation function, and 1×1 convolution to obtain the second compressed feature.

[0028] In one possible implementation, to overcome the bandwidth overhead caused by transmitting uncompressed features, this application designs a feature compression encoder and a feature recovery decoder based on dual attention and residual concatenation enhancement. The input to the feature compression encoder is the intermediate features from the second stage on the end side: The compression module employs a residual dual-branch structure. The residual branch will... Compress to: The main branch first reduces the number of channels from 192 to 48 using convolutional layers, then integrates spatial attention and channel attention mechanisms sequentially. Spatial attention calculates the mean and maximum values ​​of the channel dimensions, concatenates them, and generates a spatial attention map through a 7×7 convolution, spatially weighting the features. Channel attention uses global average pooling and global max pooling, and generates channel attention weights through a shared MLP, weighting the features by channel. The weighted features are then processed by a 3×3 depthwise separable convolution, batch normalization, ReLU, and a 1×1 convolution to obtain the main branch output: The final compressed feature is the concatenation of the outputs of the two branches, which forms the initial compressed feature. .

[0029] Furthermore, to further reduce the amount of data, a configurable b-bit quantizer is introduced. For the initial compression features Each of them The output after cropping and quantization is as follows: ,in To enhance robustness to channel variations, Gaussian noise is added. This yields the final compression features with enhanced noise: .

[0030] In some embodiments of this application, the specific process of performing high-fidelity restoration of the final compressed features to obtain the reconstructed feature map includes: expanding the number of channels of the final compressed features to 96 channels using lightweight convolution operations to obtain channel-expanded features; iteratively refining the channel-expanded features through L cascaded enhancement modules, each enhancement module including a residual refinement block, a channel attention mechanism, and a spatial attention mechanism; and restoring the iteratively refined features to the original number of channels through 1×1 convolution to obtain the reconstructed feature map.

[0031] In one possible implementation, after the edge server receives the final compressed features, the feature recovery decoder reconstructs the features step-by-step through a three-level structure. First, lightweight convolutional blocks are used to... Expand to 96 channels; then, via Each cascaded enhancement module is used for iterative refinement, and each enhancement module includes a residual refinement block, channel attention, and spatial attention. Finally, the original number of channels is restored through a 1×1 convolution to obtain the reconstructed feature map. It is then fed into the third stage of the ConvNeXt-T backbone network for further processing.

[0032] Among them, the grid-level sparse MoE module integrates a preset grid-level sparse hybrid expert model, which includes a gated network and multiple expert networks. Each expert network is implemented by a 1×1 convolution and corresponds to a learnable embedding vector. The embedding vectors of all experts constitute the expert embedding matrix.

[0033] In some embodiments of this application, the specific process of outputting inference results based on reconstructed feature maps and a preset grid-level sparse hybrid expert model includes: dividing the reconstructed feature map into spatial grids, with each grid cell serving as a basic input cell; projecting the grid features of the basic input cells onto a learnable matrix using a gated network to obtain projected features; calculating the similarity between the projected features and each expert embedding vector in the expert embedding matrix; performing norm normalization on the similarity scores; converting the normalized similarity into an expert selection probability distribution using the Softmax function; selecting and activating the K expert networks with the highest probabilities from the expert selection probability distribution; performing a 1×1 convolution output on each of the selected K experts and weighting the sum according to their corresponding gate probabilities to obtain the final output of the integrated input cell; combining the final outputs of the integrated input cell into a complete output feature map; and performing a multimodal inference task on the complete output feature map based on the third and fourth processing stages to obtain the model inference results.

[0034] In one possible implementation, to efficiently process multimodal features on an edge server, this application proposes a grid-level sparse MoE module. This module processes the reconstructed feature map into a spatial grid, with each grid cell... Dynamic routing is performed using the basic input unit. MoE is a pre-defined grid-level sparse hybrid expert model, which consists of multiple expert networks and a gated network. The expert networks are implemented using 1×1 convolutions, with each expert corresponding to a learnable embedding vector. The embedding vectors of all experts constitute the expert embedding matrix. The mesh features of the basic input unit. First, through learnable matrices Projection involves calculating the similarity between the projected features and each expert embedding, performing norm normalization, and then converting the result into an expert selection probability distribution using the Softmax function. Temperature coefficient. Control the concentration of the probability distribution. Then, Operator with the highest probability Several experts are activated and weighted according to their corresponding probabilities, while the outputs of the remaining experts are set to zero. The final output is: The gate function is defined as follows: .

[0035] It should be noted that this application integrates the MoE block into the deeper stages of the ConvNeXt-T backbone network (the [0,2,4,6,8]th block of the third stage and the [0,2]th block of the fourth stage) to make the expert network more focused on high-level semantic features. Unlike traditional image-level routing, the grid-level architecture implements finer-grained sparse activation, enabling different experts to handle different modalities, semantics, or spatial patterns separately. Sparse activation preserves modality-specific representations through expert division of labor and promotes cross-modal knowledge sharing through adaptive routing, avoiding representation interference in densely connected architectures.

[0036] In some embodiments of this application, the specific process of constructing a grid-level importance map includes: constructing a background distribution, which is either a random noise distribution or a typical input sample distribution; randomly sampling background samples from the background distribution using Monte Carlo sampling, and randomly sampling interpolation coefficients from the uniform distribution; performing linear interpolation between the interpolation coefficients and the background samples to generate interpolation points; calculating the gradient of the model output with respect to the interpolation points, repeating this process multiple times and calculating the expectation to obtain the expected gradient; calculating the Hadamard product of the expected gradient and the difference between the input feature map and the background to obtain the attribution value of each feature unit; averaging the absolute values ​​of the attribution values ​​of each feature unit along the channel dimension to obtain the unit-level aggregated value; dividing the spatial dimension of the input feature map into G×G non-overlapping grids, each grid containing a set of adjacent spatial units; averaging the unit-level aggregated values ​​of all spatial units within each grid as the importance value of that grid to obtain the grid-level importance map.

[0037] In one possible implementation, to select the most critical features for transmission under limited bandwidth, this application proposes a feature importance analysis method based on GradientSHAP and introduces a local aggregation mechanism to generate a grid-level importance map.

[0038] Specifically, the GradientSHAP method is used to calculate the importance of units. This method maintains the advantages of Shapley value theory while avoiding exponential complexity through model differentiability.

[0039] First, construct the background distribution. For example, random noise or typical input samples. Background samples are randomly selected using Monte Carlo sampling. and interpolation coefficients Generate interpolation points: .

[0040] Subsequently, the computational model outputs a pair gradient ,repeat Calculate the expected value using the union of the two factors. Final attribution value. The Hadamard product of the expected gradient and the input-background difference: Since cell-level importance is highly sensitive to noise, and making transmission decisions directly based on cell-level features would lead to high decision complexity, this application further divides the feature space into... A non-overlapping grid is used to generate a grid importance map through local aggregation. For the grid... Its importance is defined as: ,in, .in, The result is the channel average absolute value of the attribution value for cell (h,w). Represents a grid The set of cells included. The aggregated mesh importance graph suppresses noise, reduces decision dimensionality, and conforms to the principle of spatial continuity. Under bandwidth constraints... The top-k grids with the highest importance are selected for transmission. Transmission set Importance retention rate Defined as: = , . In this way, the present application can select more valuable grid features for transmission based on feature contribution, thereby preserving as much information as possible that contributes to the inference task under limited bandwidth.

[0041] In some embodiments of this application, the specific process of generating adaptive transmission decisions based on a grid-level importance graph includes: under bandwidth constraints, based on the grid-level importance graph, sorting all grids in descending order of importance value, selecting the top k grids to form a transmission set, and not transmitting the features of unselected grids; calculating the ratio of the sum of the importance values ​​of all grids in the transmission set to the sum of the importance values ​​of all grids to obtain the importance retention rate; and based on the importance retention rate, performing joint decision-making at the modality level, sample level, and feature granularity level to generate an adaptive transmission decision.

[0042] Specifically, based on the importance retention rate, the process of generating adaptive transmission decisions through joint decision-making at the modality, sample, and feature granularity levels includes: setting the system to collect multiple independent samples per second, with samples from various modalities; providing each sample with multiple feature transmission ratio options of different granularities, arranged from smallest to largest, with the largest ratio representing complete transmission; defining the bandwidth consumed by transmitting sample s using the k-th transmission ratio as the product of that transmission ratio and the bandwidth required for complete sample feature transmission; defining the importance retention rate as the proportion of task-related information retained when transmitting samples at that ratio; introducing a binary decision variable, taking one if sample s is transmitted with the k-th feature ratio, and zero otherwise; constructing a multi-choice knapsack problem with the objective function of maximizing the average feature importance retention rate of all samples, and constraints such as total bandwidth consumption not exceeding the system bandwidth limit and each sample selecting at most one transmission ratio; calculating the theoretical minimum coverage bandwidth required for all samples to be transmitted using the lowest feature ratio; and considering the available system bandwidth... When the coverage bandwidth is greater than or equal to the theoretical minimum coverage bandwidth, a coverage-first strategy is adopted: full coverage initialization is performed, the configuration with the minimum bandwidth is selected for each sample, and the remaining bandwidth is calculated; when the remaining bandwidth is greater than a preset threshold, a sample-configuration pair that maximizes marginal utility is searched cyclically, the sample is upgraded to a better configuration, and the remaining bandwidth is updated, until the traversal is completed or the remaining bandwidth is insufficient; when the available bandwidth of the system is less than the theoretical minimum coverage bandwidth, an efficiency-first strategy is adopted: for each sample-configuration pair, an efficiency index is calculated, which is the importance retention rate obtained per unit bandwidth; all sample-configuration pairs are sorted from high to low according to the efficiency index; the used bandwidth is initialized to zero and the selected sample set is an empty set; the sample-configuration pairs in the sorted list are traversed sequentially, if the sample is not selected and the current used bandwidth plus the bandwidth consumption of the option does not exceed the total bandwidth, then the option is added to the selected set, and the used bandwidth and the selected sample set are updated, until the traversal is completed; the overall feature importance retention rate and the selected set are output, and an adaptive transmission decision is generated.

[0043] In one possible implementation, an adaptive bandwidth optimization algorithm for joint selection of modalities, samples, and features is designed to dynamically decide which modalities and which samples to transmit, as well as the granularity of features for each sample, under the constraint of limited communication bandwidth, thereby improving the overall inference performance of the system.

[0044] Considering the system collects data per second Individual samples, these samples are from Different modalities. For each sample, this application provides Different feature transfer ratio options with different granularities are denoted as ,satisfy The core challenge lies in jointly optimizing modality selection, sample screening, and feature transmission granularity under bandwidth constraints to maximize the average feature importance retention rate. Let... The bandwidth required for transmitting complete sample features is represented by the [missing information - likely a specific parameter or value]. Type of transmission ratio transmission sample The bandwidth consumed is: Define importance retention rate. , indicating proportion Transmitted Samples The proportion of task-related information retained during the process is calculated by the aforementioned feature importance analysis module based on Shapley values. A binary decision variable is introduced. If the sample With the first If the characteristic is transmitted proportionally, the value is 1; otherwise, it is 0. Let... Let represent the set of all decision variables. The optimization problem is formalized as follows: ,

[0045] , , , , .

[0046] The objective function represents maximizing the average feature importance retention rate across all samples. The first constraint ensures that the total bandwidth consumption does not exceed the system bandwidth limit. The second constraint restricts each sample to selecting at most one transmission ratio; the third constraint is a binary variable constraint. This optimization problem naturally achieves mode selection by basing it on importance values. and bandwidth costs Select or discard samples from different modalities.

[0047] The search space size of this optimization problem is . , It grows exponentially. Formally, this is a multi-choice knapsack problem (MCKP): each sample corresponds to a group of items, and the group contains... Each item corresponds to a different characteristic proportion, and each item has a weight. and value Each group can select at most one item. Theoretical analysis shows that this problem is NP-hard. To solve this problem, this application proposes an adaptive bandwidth optimization algorithm, the core of which is to dynamically adjust the strategy based on bandwidth availability to obtain a near-optimal solution in polynomial time complexity.

[0048] First, calculate the theoretical minimum coverage bandwidth. This refers to the total bandwidth required to transmit all samples using the lowest feature ratio. The algorithm handles the following two cases: like A coverage-first strategy is adopted: all samples are transmitted with the lowest bandwidth configuration, and the remaining bandwidth is used to gradually upgrade the samples with the highest marginal utility. Specifically, full coverage initialization is performed first, for each sample... Select the configuration with the lowest bandwidth: To ensure all samples receive basic transmission services, the bandwidth already consumed is... The remaining bandwidth is Then it entered the iterative upgrade phase, when the remaining bandwidth... When the speed exceeds a preset threshold ε (e.g., 0.01 Mbps), repeat the following steps: find sample - configure pair. This makes marginal utility Maximize, where For the sample The current configuration; then the sample Upgrade to configuration At the same time, update the remaining bandwidth. This strategy, while ensuring that all samples receive basic services, allocates the remaining bandwidth to the samples that best improve overall performance based on the principle of maximizing marginal utility, thus achieving optimal resource allocation.

[0049] When the available bandwidth of the system cannot meet the full coverage requirement, that is In some cases, the system cannot transmit all samples with the minimum configuration. Therefore, an efficiency-first strategy must be adopted, selecting samples based on the importance-bandwidth ratio. Specifically, for each sample-configuration pair (s,k), an efficiency index is calculated. This metric represents the importance retention rate achievable per unit of bandwidth, reflecting the efficiency of bandwidth resource utilization. A greedy selection process is then performed, applying all... Each option is pressed Sort in descending order and initialize used bandwidth. Select sample set Then iterate through the sorted list; for each If the sample Not yet selected ( And the current bandwidth used plus the bandwidth consumption of this option does not exceed the total bandwidth. If the option is selected, then add it to the selected set and update it. and This selection strategy concentrates limited bandwidth resources on samples that yield the highest efficiency, thereby improving the system's inference performance under limited bandwidth conditions.

[0050] Regarding the algorithm complexity analysis, the adaptive bandwidth optimization algorithm proposed in this application first needs to calculate all The efficiency metrics for each sample-configuration pair require this step. This operation is performed once; then, these candidates are sorted in descending order of efficiency value, and the time complexity of the sorting process is O(n). When the system is in full coverage condition, that is... At that time, the remaining bandwidth is allocated through iterative upgrades, but each upgrade involves only a limited amount of additional computation, and the number of upgrades is limited. much smaller Therefore, the total complexity of this branch is still determined by the sorting steps, remaining within the range of... Quantity. When the system bandwidth At this time, the algorithm directly performs greedy selection based on the already sorted list, and the complexity is also O(n log n). In summary, the overall time complexity of the algorithm in this application, even in the worst case, is significantly lower than the exponential complexity of exhaustive search. It significantly reduces bandwidth usage and can obtain near-optimal solutions in polynomial time, making it perfectly suitable for real-time bandwidth allocation scenarios.

[0051] This algorithm is tightly integrated with the aforementioned feature importance analysis module: importance value Bandwidth consumption is provided by the feature importance analysis module based on Shapley values. The parameters are determined by the feature compression module. Through this collaborative design, this application can adaptively adjust the transmission strategy under arbitrary bandwidth constraints, achieving a trade-off between communication resources and inference accuracy.

[0052] For example Figure 2 As shown, Figure 2This is a flowchart illustrating a nonmaximum suppression method for modality, sample, and feature joint selection provided in this application. This method is used to filter candidate boxes in object detection tasks to retain the most representative detection results. The process starts at the beginning node, first inputting an importance matrix (characterizing the confidence or importance of each candidate box), a bandwidth matrix (characterizing the feature transmission cost or resource consumption of each candidate box), and a total bandwidth constraint (the maximum communication or computational resources that the system can allocate). Subsequently, the minimum coverage bandwidth is calculated, which is the total bandwidth required when all candidate boxes are configured with the lowest resource consumption. Next, the system determines whether the total bandwidth is greater than or equal to the minimum coverage bandwidth. If so, it enters the coverage-first strategy branch—first, it allocates the minimum resource configuration to each candidate box and calculates the remaining available bandwidth; if the remaining bandwidth is greater than a preset threshold, it iteratively searches for the candidate box-configuration pair that maximizes marginal utility (i.e., the one with the greatest increase in importance per unit of resource increment), upgrades its resource quota, and updates the remaining bandwidth, repeating this process until resources are exhausted or can no longer be increased; if not, it enters the efficiency-first strategy branch—it calculates the efficiency index (importance / resource consumption) for each candidate box-configuration pair, sorts all candidate box-configuration pairs in descending order of this index; initializes the used bandwidth to zero and the selected set to empty; it iterates through the sorted list sequentially, and if the current candidate box is not selected and its resource consumption plus used bandwidth does not exceed the total bandwidth, it adds it to the selected set and updates the used bandwidth, repeating this process until the traversal is complete. Finally, it outputs the overall feature importance retention rate (reflecting the amount of information retained after filtering) and the selected set (i.e., the final retained candidate boxes and their configurations), and the process ends. This process dynamically balances resource constraints and information importance to achieve optimal candidate box retention under limited bandwidth, making it suitable for scenarios such as edge-edge collaborative reasoning and multimodal feature transmission.

[0053] In this embodiment, by deploying a lightweight shared feature extraction module and a feature compression encoding module on the edge side, and deploying a feature reconstruction module and a grid-level sparse hybrid expert model on the edge server, the multimodal large model inference capability is efficiently extended to resource-constrained terminal scenarios, significantly reducing uplink communication overhead. Simultaneously, by introducing a feature importance analysis mechanism based on Shapley values, modality selection, sample screening, and feature transmission granularity are jointly modeled as a knapsack problem with grouping constraints for optimization. This enables dynamic selection of the most valuable features for transmission under bandwidth-constrained conditions, maintaining high inference performance while reducing communication overhead. This achieves an effective trade-off between communication resources and inference accuracy, providing a feasible systematic solution for the collaborative deployment of multimodal large and small models in resource-constrained networks.

[0054] Please see Figure 3 This document provides a flowchart illustrating a multimodal edge-to-edge collaboration method based on a hybrid expert model, as described in an embodiment of this application. Figure 3As shown, the detection method in this application embodiment may include the following steps: S101, a multimodal data acquisition unit collects raw multimodal data; a shared feature extractor is used to extract features from the raw multimodal data to obtain intermediate features; S102, the feature compression encoder compresses the intermediate features to obtain the final compressed features, which are then sent to the edge server; S103, the feature recovery decoder performs high-fidelity recovery of the final compressed features to obtain the reconstructed feature map; S104, the grid-level sparse MoE module outputs inference results based on reconstructed feature maps and a preset grid-level sparse hybrid expert model; S105, the feature importance analysis module constructs a grid-level importance graph, solves the problem based on the grid-level importance graph, and generates adaptive transmission decisions to guide terminal nodes in uploading features.

[0055] In this embodiment, by deploying a lightweight shared feature extraction module and a feature compression encoding module on the edge side, and deploying a feature reconstruction module and a grid-level sparse hybrid expert model on the edge server, the multimodal large model inference capability is efficiently extended to resource-constrained terminal scenarios, significantly reducing uplink communication overhead. Simultaneously, by introducing a feature importance analysis mechanism based on Shapley values, modality selection, sample screening, and feature transmission granularity are jointly modeled as a knapsack problem with grouping constraints for optimization. This enables dynamic selection of the most valuable features for transmission under bandwidth-constrained conditions, maintaining high inference performance while reducing communication overhead. This achieves an effective trade-off between communication resources and inference accuracy, providing a feasible systematic solution for the collaborative deployment of multimodal large and small models in resource-constrained networks.

[0056] This application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the multimodal end-edge collaboration method based on a hybrid expert model provided in the above-described method embodiments.

[0057] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the multimodal edge-to-edge collaboration method based on a hybrid expert model according to the various method embodiments described above.

[0058] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The multimodal edge-end collaboration program based on a hybrid expert model can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the multimodal edge-end collaboration program based on a hybrid expert model can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0059] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A multimodal edge-cooperative system based on a hybrid expert model, characterized in that, The system includes: At least one terminal node and an edge server are provided; the terminal node is equipped with a multimodal data collector, a shared feature extractor, and a feature compression encoder; the edge server is equipped with a feature recovery decoder, a grid-level sparse MoE module, and a feature importance analysis module; wherein... The multimodal data acquisition device is used to acquire raw multimodal data; the shared feature extractor is used to extract features from the raw multimodal data to obtain intermediate features. The feature compression encoder is used to compress the intermediate features to obtain the final compressed features, which are then sent to the edge server. The feature recovery decoder is used to perform high-fidelity recovery of the final compressed features to obtain a reconstructed feature map; The grid-level sparse MoE module is used to output inference results based on the reconstructed feature map and a preset grid-level sparse hybrid expert model. The feature importance analysis module is used to construct a grid-level importance graph, solve the problem based on the grid-level importance graph, and generate an adaptive transmission decision to guide the terminal node in uploading features.

2. The system according to claim 1, characterized in that, The shared feature extractor is implemented based on the ConvNeXt-T backbone network, which includes a first processing stage, a second processing stage, a third processing stage, and a fourth processing stage arranged sequentially. The first and second processing stages are deployed on the terminal nodes. The third and fourth processing stages are deployed on the edge servers and integrated with the grid-level sparse MoE module. The first processing stage and the second processing stage are used to perform shallow feature extraction on the original multimodal data and output intermediate features; The third and fourth processing stages are used to receive the reconstructed feature map output by the feature recovery decoder and perform sparse expert inference based on the reconstructed feature map and the grid-level sparse MoE module.

3. The system according to claim 1, wherein the feature compression encoder adopts a residual dual-branch structure, the residual dual-branch structure including a residual branch and a main branch; The step of compressing the intermediate features to obtain the final compressed features and sending them to the edge server includes: The residual branch compresses the intermediate features to obtain the first compressed feature; The main branch sequentially integrates spatial attention mechanism and channel attention mechanism for weighted processing of the intermediate features to obtain the second compressed feature; The first compression feature is concatenated with the second compression feature to obtain the initial compression feature; For each element in the initial compressed feature, after pruning and quantization, the quantized feature is output. Gaussian noise is added to the quantized features to obtain the final compressed features with enhanced noise. The final compressed feature of the noise enhancement is sent to the edge server via a wireless link.

4. The system according to claim 3, characterized in that, The intermediate features are sequentially integrated with spatial attention and channel attention mechanisms for weighted processing to obtain the second compressed feature, including: The number of channels in the intermediate features is reduced by using convolutional layers to obtain dimensionality-reduced features; Spatial attention and channel attention mechanisms are applied to the dimensionality reduction features to obtain features with dual attention weighting. The spatial attention mechanism calculates the mean and maximum values ​​of the feature map in the channel dimension for the dimensionality reduction features, and concatenates the two values ​​and then performs a 7×7 convolution to generate a spatial attention map. The channel attention mechanism extracts channel descriptors for the dimensionality reduction features through global average pooling and global max pooling, and then generates channel attention weights through a shared MLP. The features weighted by dual attention are sequentially passed through a 3×3 depthwise separable convolution, batch normalization, ReLU activation function, and 1×1 convolution to obtain the second compressed feature.

5. The system according to claim 1, characterized in that, The step of performing high-fidelity restoration on the final compressed features to obtain a reconstructed feature map includes: The number of channels in the final compressed feature is expanded to 96 channels using a lightweight convolution operation to obtain the channel-expanded feature; The channel extended features are iteratively refined through L cascaded enhancement modules, each of which includes a residual refinement block, a channel attention mechanism, and a spatial attention mechanism. The features refined through iterative refinement are restored to the original number of channels using 1×1 convolution, resulting in a reconstructed feature map.

6. The system according to claim 2, characterized in that, The grid-level sparse MoE module integrates a preset grid-level sparse hybrid expert model, which includes a gated network and multiple expert networks. Each expert network is implemented by a 1×1 convolution and corresponds to a learnable embedding vector. The embedding vectors of all experts constitute the expert embedding matrix. Based on the reconstructed feature map and the preset grid-level sparse hybrid expert model, the inference results are output, including: The reconstructed feature map is divided into a spatial grid, and each grid cell in the spatial grid serves as a basic input cell. For the grid features of the basic input unit, the gated network projects them through a learnable matrix to obtain projected features; the similarity between the projected features and each expert embedding vector in the expert embedding matrix is ​​calculated; the similarity scores are normalized using norm normalization; the normalized similarity is converted into an expert selection probability distribution using the Softmax function; the K expert networks with the highest probabilities are selected and activated from the expert selection probability distribution; the 1×1 convolution outputs of the selected k experts are weighted and summed according to their corresponding gating probabilities to obtain the final output of the integrated input unit; the final outputs of the integrated input unit are combined into a complete output feature map. Based on the third and fourth processing stages, a multimodal reasoning task is performed on the complete output feature map to obtain the model reasoning result.

7. The system according to claim 1, characterized in that, Constructing a grid-level importance graph includes: Construct a background distribution, which is either a random noise distribution or a typical input sample distribution; Monte Carlo sampling is used to randomly extract background samples from the background distribution and randomly extract interpolation coefficients from the uniform distribution. Linear interpolation is then performed between the interpolation coefficients and the background samples to generate interpolation points. The gradient of the model output with respect to the interpolation point is calculated, and this process is repeated multiple times to obtain the expected gradient. Calculate the Hadamard product of the expected gradient and the input feature map-background difference to obtain the attribution value for each feature unit; The unit-level aggregated value is obtained by averaging the absolute values ​​of the attribution values ​​of each feature unit along the channel dimension. The spatial dimension of the input feature map is divided into G×G non-overlapping grids. Each grid contains a set of adjacent spatial cells. The average value of the cell-level aggregated value of all spatial cells in each grid is taken as the importance value of that grid, thus obtaining the grid-level importance map.

8. The system according to claim 1, characterized in that, The process of solving based on the grid-level importance graph to generate adaptive transmission decisions includes: Under bandwidth constraints, based on the grid-level importance graph, all grids are sorted in descending order of importance value, and the top k grids are selected to form a transmission set. Features of grids not selected are not transmitted. The importance retention rate is obtained by calculating the ratio of the sum of importance values ​​of all grids in the transmission set to the sum of the importance values ​​of all grids. Based on the importance retention rate, joint decision-making is performed at the modality level, sample level, and feature granularity level to generate an adaptive transmission decision.

9. The system according to claim 8, characterized in that, The process of generating adaptive transmission decisions based on the importance retention rate, through joint decision-making at the modality, sample, and feature granularity levels, includes: The system is configured to collect multiple independent samples per second, which come from various modalities. For each sample, multiple feature transmission ratio options with different granularities are provided, arranged from smallest to largest, with the largest ratio representing complete transmission. The bandwidth consumed by transmitting sample s using the k-th transmission ratio is defined as the product of that ratio and the bandwidth required for complete sample feature transmission. The importance retention rate is defined as the proportion of task-related information retained when transmitting samples at that ratio. A binary decision variable is introduced: if sample s is transmitted using the k-th feature ratio, the value is 1; otherwise, it is 0. With the objective function of maximizing the average feature importance retention rate of all samples, and with the constraints that the total bandwidth consumption does not exceed the system bandwidth limit and each sample can choose at most one transmission ratio, a multi-choice knapsack problem is constructed. Calculate the theoretical minimum coverage bandwidth required to transmit all samples using the lowest feature ratio; When the available bandwidth of the system is greater than or equal to the theoretical minimum coverage bandwidth, a coverage priority strategy is adopted: full coverage initialization is performed, the configuration with the minimum bandwidth is selected for each sample, and the remaining bandwidth is calculated; when the remaining bandwidth is greater than the preset threshold, the sample-configuration pair is searched in a loop to maximize the marginal utility, the sample is upgraded to a better configuration, and the remaining bandwidth is updated until the traversal is completed or the remaining bandwidth is insufficient. When the available bandwidth of the system is less than the theoretical minimum coverage bandwidth, an efficiency-first strategy is adopted: For each sample-configuration pair, an efficiency index is calculated, which is the importance retention rate that can be obtained per unit bandwidth; all sample-configuration pairs are sorted from high to low according to the efficiency index; the used bandwidth is initialized to zero and the selected sample set is empty; the sample-configuration pairs in the sorted list are traversed sequentially. If the sample is not selected and the current used bandwidth plus the bandwidth consumption of the option does not exceed the total bandwidth, the option is added to the selected set, and the used bandwidth and the selected sample set are updated until the traversal is completed; Output the overall feature importance retention rate and the selected set to generate an adaptive transmission decision.

10. A multimodal edge collaboration method based on a hybrid expert model implemented using the method described in any one of claims 1-9, characterized in that, The method includes: A multimodal data acquisition device collects raw multimodal data; the shared feature extractor is used to extract features from the raw multimodal data to obtain intermediate features; The feature compression encoder compresses the intermediate features to obtain the final compressed features, which are then sent to the edge server. The feature recovery decoder performs high-fidelity recovery on the final compressed features to obtain a reconstructed feature map; The grid-level sparse MoE module outputs inference results based on the reconstructed feature map and a preset grid-level sparse hybrid expert model. The feature importance analysis module constructs a grid-level importance graph, solves the graph based on the grid-level importance graph, and generates an adaptive transmission decision to guide the terminal node in uploading features.

Images