High-risk off-network multi-modal large model lightweight edge collaborative inference method and system

Abstract

The application discloses a high-risk off-network multi-modal large model lightweight edge collaborative reasoning method and system, which comprises the following steps: performing operator-level splitting on a pre-trained visual language model, and constructing a global task dependency graph; determining the effective computing power of each edge computing power node in the self-organizing network, combining the global task dependency graph, and determining the computing power fingerprint of each edge computing power node; task decomposition is performed on the global task dependency graph to obtain a lightweight task operator dependency subgraph, and the computing power node allocation scheme is determined in combination with the computing power fingerprint of each edge computing power node; an operator allocation request is sent to each edge computing power node, and each edge computing power node loads necessary model parameters according to the operator allocation request; the pipeline mode is adopted to sequentially execute operator calculation through each edge computing power node; each edge computing power node transmits the calculation vector through the self-organizing network, verifies the calculation result, and outputs the final consensus result. The application improves the continuous service capability in the high-risk off-network scene.

Description

Technical Field

[0001] This application relates to the field of edge computing technology, and in particular to a lightweight edge collaborative inference method and system for high-risk off-grid multimodal large models. Background Technology

[0002] In recent years, with the rapid development of unmanned systems, intelligent equipment, edge AI, and Vision Language Models (VLMs), natural language-driven multi-agent collaborative decision-making technology has been increasingly applied to complex scenarios such as emergency rescue and unmanned inspection. Compared to traditional manually preset rules, VLMs can achieve a unified understanding of multimodal information such as images and text, and generate intelligent decision-making results by combining natural language tasks. However, these scenarios are usually located in network environments with insufficient network infrastructure or limited communication, such as mountainous areas and underground spaces. Furthermore, the edge devices involved in the collaboration generally suffer from limitations in computing power, storage, and energy, making it difficult to directly deploy large models on edge nodes and complete real-time collaborative inference.

[0003] Currently, to address the challenge of running large models on edge devices, existing technologies typically employ edge-cloud collaborative inference and distributed inference solutions. These solutions deploy computationally intensive models in the cloud, with edge devices handling only data collection and some front-end inference. However, edge-cloud collaborative inference relies on stable network connections and cannot utilize cloud computing power for model inference in off-grid environments, resulting in poor system real-time performance, reliability, and environmental adaptability. Summary of the Invention

[0004] The main purpose of this application is to provide a lightweight edge collaborative reasoning method and system for high-risk off-grid multimodal large models, aiming to solve the technical problem of poor environmental adaptability of edge-cloud collaborative reasoning in related technologies.

[0005] To achieve the above objectives, this application proposes a lightweight edge collaborative reasoning method for high-risk off-grid multimodal large models, the method comprising: The pre-trained visual language model is decomposed at the operator level to construct a global task dependency graph; Periodically check the edge node computing power pool to determine the effective computing power of each edge computing power node in the self-organizing network, and combine it with the global task dependency graph to determine the computing power fingerprint of each edge computing power node; The global task dependency graph is decomposed to obtain a lightweight task operator dependency subgraph. Combined with the computing power fingerprint of each edge computing power node, the computing power node allocation scheme is determined. Send operator allocation requests to each edge computing node, so that each edge computing node can load the necessary model parameters according to the operator allocation requests; Initiate edge collaborative reasoning and use a pipeline approach to sequentially execute operator calculations through each edge computing node; Each edge computing node transmits computing vectors through a self-organizing network, verifies the computing results with each other through a decentralized consensus algorithm, and outputs the final consensus result.

[0006] In one embodiment, the step of constructing a global task dependency graph by decomposing a pre-trained visual language model at the operator level includes: The pre-trained visual language model is decomposed layer by layer into indivisible atomic operators; Based on the data flow dependencies of each atomic operator in the forward inference process, a directed acyclic graph is constructed as the initial global task dependency graph; wherein, the vertices of the directed acyclic graph correspond to atomic operators, and the directed edges represent the outputs of the upstream operators that constitute the inputs of the downstream operators; Operator fusion is performed on adjacent atomic operators that satisfy the fusion condition in the initial global task dependency graph to obtain the global task dependency graph.

[0007] In one embodiment, the steps of periodically detecting the edge node computing power pool, determining the effective computing power of each edge computing power node in the ad hoc network, and determining the computing power fingerprint of each edge computing power node by combining the global task dependency graph include: Send probe requests to each edge computing node in the self-organizing network at a preset detection cycle to obtain the computing power status information returned by each edge computing node; Based on the operator type and tensor shape of the operators corresponding to each vertex in the global task dependency graph, a representative set of operators is determined; the representative set of operators is the set of operators obtained after deduplicating the operator types and tensor shapes in the global task dependency graph. For each edge computing node, benchmark tests are performed on each representative operator in the representative operator set, and the computing power consumption characteristics of each edge computing node executing each representative operator are recorded. The computing power status characteristics of each edge computing node are associated and encapsulated with the computing power occupancy characteristics of its corresponding representative operators to generate the computing power fingerprint of each edge computing node.

[0008] In one embodiment, the steps of decomposing the global task dependency graph to obtain a lightweight task operator dependency subgraph, and determining the computing power node allocation scheme by combining the computing power fingerprints of each edge computing power node include: The global task dependency graph is divided according to a preset decomposition strategy to obtain multiple lightweight task operator dependency subgraphs; Determine the computational cost characteristics and cross-subgraph communication characteristics of each lightweight task operator-dependent subgraph; the computational cost characteristics represent the cumulative computational cost of each atomic operator within the lightweight task operator-dependent subgraph, and the cross-subgraph communication characteristics represent the data transfer cost between the lightweight task operator-dependent subgraph and adjacent lightweight task operator-dependent subgraphs. The computational characteristics of each lightweight task operator dependency subgraph are matched with the computational fingerprint of each edge computing node to determine the fit between each lightweight task operator dependency subgraph and each edge computing node. Based on the cross-subgraph communication traffic characteristics of each lightweight task operator-dependent subgraph and the link communication capabilities between edge computing nodes in the ad hoc network, the inter-subgraph communication cost matrix is ​​determined. Based on the computational fit between each lightweight task operator-dependent subgraph and each edge computing node, as well as the communication cost matrix between subgraphs, a computing node allocation scheme is determined with the goal of minimizing global inference latency.

[0009] In one embodiment, the step of segmenting the global task dependency graph according to a preset decomposition strategy to obtain multiple lightweight task operator dependency subgraphs includes: Assign a communication cost weight to each directed edge in the global task dependency graph, and an computation cost weight to each vertex; The path with the highest cumulative computational cost weight in the global task dependency graph is identified as the critical path. The critical path is then divided equally according to the cumulative computational cost weight to determine the initial set of splitting edges. Based on the initial set of splitting edges, the global task dependency graph is divided into multiple initial subgraphs to obtain the initial splitting scheme. Perform iterative refinement on the initial segmentation scheme, and calculate the sum of the communication cost weights of all directed edges across the subgraph in the current segmentation scheme as the current segmentation cost; Traverse all vertices located on the boundaries of the subgraph and calculate the change in segmentation cost after migrating each boundary vertex to its adjacent subgraph. Select the boundary vertices that maximize the reduction in segmentation cost to perform migration, and ensure that the sum of the computational cost weights of each subgraph after migration satisfies the preset equilibrium constraint. Repeatedly perform iterative refinement operations until there are no legal migration operations that reduce the cutting cost and satisfy the preset equilibrium constraints. Finally, the multiple initial subgraphs obtained are used as multiple lightweight task operator dependency subgraphs.

[0010] In one embodiment, the step of sending an operator allocation request to each edge computing node, so that each edge computing node loads the necessary model parameters according to the operator allocation request, includes: Based on the computing power node allocation scheme, determine the operator allocation list corresponding to each edge computing power node; Based on the operator allocation list of each edge computing node, generate corresponding operator allocation requests, and send each operator allocation request to the corresponding edge computing node through the self-organizing network; For each edge computing node, based on the parameter addressing information of each atomic operator in the received operator allocation request, the model parameter fragments of the corresponding atomic operator are retrieved from the locally pre-stored model parameter library and loaded. For each edge computing node, perform parameter integrity verification on each loaded model parameter segment and generate verification results; If the verification results of each edge computing node indicate that the verification has passed, each edge computing node returns to the loading ready state to trigger the step of starting edge collaborative reasoning.

[0011] In one embodiment, the steps of initiating edge collaborative inference and sequentially executing operator calculations through each edge computing node in a pipelined manner include: Based on the mapping relationship between each lightweight task operator dependency subgraph and each edge computing node in the computing node allocation scheme, and the data flow dependency order between each lightweight task operator dependency subgraph, the pipeline execution sequence is determined. The input data to be inferred is divided into multiple inference micro-batches according to a preset micro-batch granularity; each inference micro-batch independently completes inference through a pipeline execution sequence. For each edge computing node, the forward computation of each assigned operator is executed sequentially according to the execution timing in the pipeline execution sequence to generate the computation vector corresponding to the current operator. After completing the forward computation of the current atomic operator, each edge computing node will pass the computation vector through the ad hoc network to the next edge computing node in the pipeline execution sequence, so as to trigger the next edge computing node to execute the forward computation of subsequent operators. After the last edge computing node in the pipeline execution sequence completes the forward computation of the last operator, the final computation vector is used as the inference output of the corresponding inference micro-batch. The inference outputs of each inference micro-batch are then summed to obtain the complete inference result.

[0012] In one embodiment, the steps of each edge computing node passing computation vectors through a self-organizing network, mutually verifying computation results through a decentralized consensus algorithm, and outputting the final consensus result include: For the first edge computing node in the pipeline execution sequence, after completing the forward computation of the current operator, the generated computation vector is passed to the next edge computing node in the pipeline execution sequence through the ad hoc network, and the computation vector is cached locally as the output vector; For each edge computing node in the pipeline execution sequence except the first edge computing node, after receiving the computing vector transmitted by the previous edge computing node and completing the forward calculation of the current operator, the generated computing vector is associated and encapsulated with the received computing vector to generate an authentication return packet, which is then transmitted back to the previous edge computing node through the self-organizing network. For each edge computing node, the authentication return packet sent back by the next edge computing node in the pipeline execution sequence is received, the received computation vector in the authentication return packet is extracted, and the consistency is compared with the locally cached output vector to generate the authentication result. For each edge computing node in the pipeline execution sequence except the first edge computing node, after receiving the authentication confirmation sent by the previous edge computing node, the locally generated computing vector is transmitted to the next edge computing node in the pipeline execution sequence through the ad hoc network, and the computing vector is cached locally as the output vector; If the last edge computing node in the pipeline execution sequence completes the forward computation of the last operator and the authentication results of each hop are consistent, the complete inference result will be output as the final consensus result.

[0013] In one embodiment, the method further includes: If an abnormal computing power is detected in the target edge computing power node, the computing power fingerprint of the target edge computing power node is obtained as the fault computing power fingerprint, and the set of idle edge computing power nodes in the idle state is determined among the other edge computing power nodes in the ad hoc network. When the set of idle edge computing power nodes is not empty, calculate the fingerprint similarity between the computing power fingerprint of each idle edge computing power node and the fingerprint of the faulty computing power, and select the idle edge computing power node with the highest fingerprint similarity as the takeover node. When the set of idle edge computing nodes is empty, calculate the fingerprint similarity between the computing power fingerprint of each edge computing node currently performing operator calculation and the fingerprint of the faulty computing node in the self-organizing network, and determine the takeover node. Send a takeover request to the takeover node. After completing the operator computation it is currently executing, the takeover node releases its current computing resources and retrieves and loads the model parameter fragments corresponding to the target edge computing node from its local pre-stored model parameter library according to the takeover request. The takeover node takes over the execution position of the target edge computing node in the pipeline execution sequence, executes the operator calculation corresponding to the target edge computing node, and continues to pass the generated calculation vector to the next edge computing node according to the pipeline execution sequence; Fault isolation is performed on the target edge computing nodes, removing them from the pipeline execution sequence, and the computing node allocation scheme is updated based on the takeover node.

[0014] Secondly, to achieve the above objectives, this application further provides a lightweight edge collaborative reasoning system for high-risk offline multimodal large-scale models, characterized in that it applies the above-mentioned lightweight edge collaborative reasoning method for high-risk offline multimodal large-scale models, and the system includes: The graph construction module performs operator-level decomposition on the pre-trained visual language model to construct a global task dependency graph; The computing power determination module is used to periodically detect the computing power pool of edge nodes, determine the effective computing power of each edge computing power node in the ad hoc network, and determine the computing power fingerprint of each edge computing power node by combining the global task dependency graph. The task allocation module is used to decompose the global task dependency graph to obtain a lightweight task operator dependency subgraph, and combine the computing power fingerprint of each edge computing power node to determine the computing power node allocation scheme. The parameter loading module is used to send operator allocation requests to each edge computing node, so that each edge computing node can load the necessary model parameters according to the operator allocation request. The collaborative reasoning module is used to initiate edge collaborative reasoning, which uses a pipeline approach to execute operator calculations sequentially through each edge computing node. The result output module is used for each edge computing node to pass computation vectors through a self-organizing network, verify the computation results with each other through a decentralized consensus algorithm, and output the final consensus result.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application deconstructs a large model into lightweight computing units that can be executed independently on resource-constrained edge devices by decomposing a pre-trained visual language model at the operator level and constructing a global task dependency graph. This fundamentally eliminates the dependence on cloud computing power, enabling the multimodal large model to still possess complete inference capabilities in off-grid environments such as mountainous areas and underground spaces where network infrastructure is lacking. By periodically detecting the effective computing power of each edge computing node and generating computing power fingerprints, and combining the task decomposition results of the global task dependency graph to perform joint optimization of computational fit matching and communication costs, precise allocation and load balancing of heterogeneous edge resources are achieved, effectively improving the overall efficiency of multi-node collaborative inference. The system achieves high efficiency by driving each edge computing node to execute operator calculations sequentially through a pipelined approach. A decentralized consensus algorithm autonomously authenticates calculation results hop-by-hop between adjacent nodes, ensuring the correctness and consistency of inference results without the need for central node arbitration. This significantly enhances the system's reliability and environmental adaptability under harsh conditions such as limited communication and dynamic node changes. Furthermore, through computing power anomaly detection and an automatic takeover mechanism based on computing power fingerprint similarity, the system can quickly select the optimal replacement node for seamless takeover when an edge node fails, avoiding single-point failures that could interrupt inference tasks. This further improves the system's robustness and continuous service capabilities in high-risk off-network scenarios. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the lightweight edge collaborative inference method for high-risk off-grid multimodal large models in the embodiments of this application.

[0019] Figure 2 This is a schematic diagram of the structure of a lightweight edge collaborative inference system for high-risk off-grid multimodal large models.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is as follows: First, the pre-trained visual language model is decomposed into atomic operators layer by layer according to the computational layer, and a global task dependency graph in the form of a directed acyclic graph is constructed. Then, the computing power status of each edge computing power node in the ad hoc network is detected at a preset period and representative operator benchmark tests are performed to generate computing power fingerprints that characterize the heterogeneous computing capabilities of each node. Subsequently, the global task dependency graph is initially segmented along the critical path and refined by iterative boundary vertex migration to obtain multiple lightweight task operator dependency subgraphs. The adaptation matching and communication cost joint optimization are performed by combining the computational volume characteristics of each subgraph, the cross-subgraph communication volume characteristics, and the computing power fingerprints of each node. The computing power node allocation scheme is determined with the goal of minimizing the global inference latency. Each edge computing power node loads the corresponding model parameters from the local model parameter library as needed. After data sharding and integrity verification, the system enters a ready state. During the inference phase, the input data is segmented at the micro-batch granularity, and each node is driven sequentially to perform forward computation in a pipeline manner. Adjacent nodes transmit computation vectors hop-by-hop through authentication return packets, and the upstream node performs a consistency comparison with its local cache. Only after successful authentication can the computation vector be triggered to continue downstream. The final node outputs the final consensus result after completing all operator computations and achieving consistent authentication across hops. Simultaneously, the system continuously monitors the operating status of each node. When an abnormal computing power is detected, the optimal takeover node is selected from idle or currently computing nodes based on computing power fingerprint similarity. After the takeover node completes its current operator, it takes over from the faulty node to continue pipelined inference, thereby achieving efficient, reliable, and fault-tolerant collaborative inference of multimodal large models on heterogeneous edge devices in an off-grid environment. Specifically, this application provides a lightweight edge collaborative inference method for high-risk off-grid multimodal large models, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the lightweight edge collaborative reasoning method for high-risk off-grid multimodal large models in this application.

[0024] In this embodiment, the lightweight edge collaborative inference method for high-risk offline multimodal large models includes steps S10~S60: Step S10: Perform operator-level decomposition on the pre-trained visual language model to construct a global task dependency graph.

[0025] Step S20: Periodically detect the edge node computing power pool, determine the effective computing power of each edge computing power node in the self-organizing network, and determine the computing power fingerprint of each edge computing power node by combining the global task dependency graph.

[0026] Step S30: Decompose the global task dependency graph to obtain a lightweight task operator dependency subgraph, and combine the computing power fingerprints of each edge computing power node to determine the computing power node allocation scheme.

[0027] Step S40: Send operator allocation requests to each edge computing node so that each edge computing node can load the necessary model parameters according to the operator allocation requests.

[0028] Step S50: Start edge collaborative reasoning and use a pipeline approach to execute operator calculations sequentially through each edge computing node.

[0029] In step S60, each edge computing node transmits computing vectors through a self-organizing network, verifies the computing results with each other through a decentralized consensus algorithm, and outputs the final consensus result.

[0030] It should be noted that the lightweight edge collaborative inference method for high-risk off-grid multimodal large models provided in this embodiment is applicable to off-grid environments such as mountainous areas and underground spaces where network infrastructure is lacking. In such environments, edge devices participating in collaboration generally suffer from limited computing power, storage, and energy, and cannot rely on cloud computing power to complete model inference.

[0031] This embodiment enables a large multimodal model to retain its complete inference capabilities even when offline by performing operator-level decomposition and distributed collaborative execution of the visual language model within an ad hoc network. Here, the visual language model refers to a pre-trained large model capable of simultaneously understanding multimodal inputs such as images and text and generating natural language decision outputs; the ad hoc network refers to a point-to-point communication network autonomously established by each edge computing node without a central infrastructure; and edge computing nodes refer to edge devices within the ad hoc network that possess certain computing capabilities and can independently execute some model inference tasks.

[0032] Specifically, operator-level decomposition refers to breaking down the overall computational process of a visual language model into indivisible minimum computational units, i.e., atomic operators, layer by layer. Each atomic operator corresponds to an independent forward computation operation within the model. The global task dependency graph is a directed acyclic graph constructed with each atomic operator as a vertex and the data flow dependencies between operators as directed edges. The directed edges represent that the output of the upstream operator constitutes the input of the downstream operator. By constructing the global task dependency graph, the complete inference process, originally coupled within a single model, is explicitly transformed into a set of discrete computational tasks that can be independently scheduled and allocated, providing a structured descriptive foundation for subsequent task decomposition and node allocation.

[0033] The edge node computing power pool is periodically tested to determine the effective computing power of each edge computing node in the ad hoc network. Combined with the global task dependency graph, the computing power fingerprint of each edge computing node is determined. The edge node computing power pool refers to the set of available computing resources for all edge computing nodes in the ad hoc network. Effective computing power refers to the actual computing capacity that each edge computing node can use to execute inference tasks at the current moment, which changes dynamically due to factors such as the node's current load, remaining memory, and energy status. The computing power fingerprint is a structured representation of the computing power of each edge computing node. It is obtained by combining the types and tensor shapes of each operator in the global task dependency graph and performing benchmark tests on representative operators on each node, reflecting the actual execution efficiency of each node for different types of operators. Since the hardware architecture, chip model, and resource reserves of each edge computing node often differ significantly in an off-grid environment, the computing power fingerprint provides a quantitative measurement basis for heterogeneous resources in subsequent task allocation.

[0034] A lightweight task operator dependency subgraph refers to a subgraph segmented from the global task dependency graph, whose computational scale is adapted to the carrying capacity of a single edge computing node. Each subgraph contains a set of atomic operators with data flow dependencies, and can be independently computed by a single node. The computing node allocation scheme refers to the mapping relationship between each lightweight task operator dependency subgraph and each edge computing node. Its determination process comprehensively considers the compatibility between the computational characteristics of each subgraph and the computing power fingerprint of each node, as well as the cost of cross-node communication between subgraphs, with the optimization goal of minimizing global inference latency. The core inventive concept of this step is to bidirectionally match the model splitting granularity with the actual computing power of the nodes. This avoids a single node becoming an inference bottleneck due to carrying too many operators, and reduces the communication overhead caused by cross-node data transmission, thereby achieving optimal overall inference efficiency in a heterogeneous off-grid environment.

[0035] The operator allocation request is an instruction message carrying a list of atomic operators to be executed by each edge computing node and their parameter addressing information; model parameters refer to the numerical parameters such as weights and biases required by each atomic operator during the forward computation process. Since the complete parameter scale of the visual language model far exceeds the storage capacity of a single edge computing node, in this step, each node only loads the model parameter fragments corresponding to the operator allocated to it according to the operator allocation request, rather than loading the complete model parameters. This achieves distributed storage and on-demand loading of model parameters within the ad hoc network, significantly reducing the storage footprint of each node.

[0036] A pipeline refers to a system where edge computing nodes are arranged sequentially according to the data flow dependencies in the global task dependency graph. After each node completes the forward computation of its assigned operator, it passes the generated computation vector to the next edge computing node in the pipeline, which then continues to execute the computation of subsequent operators. The computation vector refers to the intermediate result tensor output by each atomic operator's forward computation, which is passed between nodes as input to downstream operators. The core advantage of using a pipelined approach is that it reduces the computational power requirement when the input data is divided into multiple inference micro-batches. Furthermore, different nodes in the pipeline can be in computation state simultaneously, allowing for parallel utilization of the computational resources of each edge computing node and effectively shortening the overall completion time of multi-batch inference.

[0037] Decentralized consensus algorithms refer to protocol mechanisms that autonomously verify the correctness of computation results among adjacent edge computing nodes without relying on a central arbitration node. Specifically, each edge computing node, while transmitting computation vectors, verifies the correctness of each hop's computation result by having the upstream node perform a consistency comparison of the computation vectors returned by the downstream node through hop-by-hop authentication. At the end of the pipeline, when the edge computing node completes the forward computation of the last operator and all hop authentication results are consistent, the complete inference result is output as the final consensus result. The inventive concept behind this step is that, in an off-network environment lacking a trusted central node, autonomous mutual verification between nodes ensures the reliability of inference results, preventing erroneous results from being silently propagated downstream due to single-point computation errors or communication interference, thereby improving the reliability of the system's results under harsh communication conditions.

[0038] In one feasible implementation, step S10 includes steps A10 to A30: Step A10: Decompose the pre-trained visual language model into indivisible atomic operators layer by layer according to the computational layer.

[0039] Step A20: Based on the data flow dependencies of each atomic operator in the forward inference process, a directed acyclic graph is constructed as the initial global task dependency graph; wherein, the vertices of the directed acyclic graph correspond to atomic operators, and the directed edges represent the outputs of upstream operators constituting the inputs of downstream operators.

[0040] Step A30: Operator fusion is performed on adjacent atomic operators that meet the fusion conditions in the initial global task dependency graph to obtain the global task dependency graph.

[0041] Step S20 includes steps B10 to B40: Step B10: Send probe requests to each edge computing node in the ad hoc network at a preset detection cycle to obtain the computing power status information returned by each edge computing node.

[0042] Step B20: Determine the representative operator set based on the operator type and tensor shape of the operators corresponding to each vertex in the global task dependency graph; the representative operator set is the set of operators obtained after deduplicating the operator types and tensor shapes in the global task dependency graph.

[0043] Step B30: For each edge computing node, perform benchmark tests on each representative operator in the representative operator set one by one, and record the computing power occupancy characteristics of each edge computing node when executing each representative operator.

[0044] Step B40: Associate and encapsulate the computing power status characteristics of each edge computing power node with the computing power occupancy characteristics of its corresponding representative operators to generate the computing power fingerprint of each edge computing power node.

[0045] Step S30 includes steps C10 to C50: Step C10: The global task dependency graph is divided according to a preset decomposition strategy to obtain multiple lightweight task operator dependency subgraphs.

[0046] Step C20: Determine the computational cost characteristics and cross-subgraph communication characteristics of each lightweight task operator dependent subgraph. The computational cost characteristics represent the cumulative computational cost of each atomic operator within the lightweight task operator dependent subgraph, and the cross-subgraph communication characteristics represent the data transfer cost between the lightweight task operator dependent subgraph and adjacent lightweight task operator dependent subgraphs.

[0047] Step C30: Match the computational complexity features of each lightweight task operator dependency subgraph with the computational power fingerprint of each edge computing node to determine the fit between each lightweight task operator dependency subgraph and each edge computing node.

[0048] Step C40: Determine the inter-subgraph communication cost matrix based on the cross-subgraph communication characteristics of each lightweight task operator-dependent subgraph and the link communication capabilities between each edge computing node in the ad hoc network.

[0049] Step C50: Based on the computational fit between each lightweight task operator dependency subgraph and each edge computing node, as well as the communication cost matrix between subgraphs, a computing node allocation scheme is determined with the goal of minimizing global inference latency.

[0050] Step C10 includes steps C11 to C16: Step C11: Assign communication cost weights to each directed edge in the global task dependency graph and computation cost weights to each vertex.

[0051] Step C12: Determine the path with the largest cumulative calculated cost weight in the global task dependency graph as the critical path, divide the critical path equally according to the cumulative calculated cost weight, determine the initial splitting edge set, and divide the global task dependency graph into multiple initial subgraphs according to the initial splitting edge set to obtain the initial splitting scheme.

[0052] Step C13: Perform an iterative refinement operation on the initial segmentation scheme and calculate the sum of the communication cost weights of all directed edges across the subgraph in the current segmentation scheme as the current segmentation cost.

[0053] Step C14: Traverse all vertices located on the boundaries of the subgraph and calculate the change in segmentation cost after migrating each boundary vertex to its adjacent subgraph.

[0054] Step C15: Select the boundary vertex that maximizes the reduction in segmentation cost and perform migration, and ensure that the sum of the computational cost weights of each subgraph after migration satisfies the preset equilibrium constraint.

[0055] Step C16: Repeat the iterative refinement operation until there is no legal migration operation that reduces the cut cost and satisfies the preset equilibrium constraint. Finally, the multiple initial subgraphs obtained are used as multiple lightweight task operator dependency subgraphs.

[0056] Step S40 includes steps D10 to D50: Step D10: Determine the operator allocation list corresponding to each edge computing node according to the computing power node allocation scheme.

[0057] Step D20: Generate corresponding operator allocation requests based on the operator allocation list of each edge computing node, and send each operator allocation request to the corresponding edge computing node through the self-organizing network.

[0058] Step D30: For each edge computing node, based on the parameter addressing information of each atomic operator in the received operator allocation request, retrieve and load the model parameter fragments of the corresponding atomic operator from the locally pre-stored model parameter library.

[0059] Step D40: For each edge computing node, perform parameter integrity verification on each loaded model parameter shard and generate verification results.

[0060] In step D50, if the verification results of each edge computing node indicate that the verification has passed, each edge computing node returns to the loading ready state to trigger the step of starting edge collaborative reasoning.

[0061] Step S50 includes steps E10 to E50: Step E10: Determine the pipeline execution sequence based on the mapping relationship between each lightweight task operator dependency subgraph and each edge computing node in the computing node allocation scheme, as well as the data flow dependency order between each lightweight task operator dependency subgraph.

[0062] Step E20: The input data to be inferred is divided into multiple inference micro-batches according to a preset micro-batch granularity; each inference micro-batch independently completes inference through the pipeline execution sequence.

[0063] Step E30: For each edge computing node, according to the execution timing in the pipeline execution sequence, the forward computation of each assigned operator is executed sequentially to generate the computation vector corresponding to the current operator.

[0064] In step E40, after each edge computing node completes the forward computation of the current atomic operator, it transmits the computation vector through the ad hoc network to the next edge computing node in the pipeline execution sequence, so as to trigger the next edge computing node to execute the forward computation of subsequent operators.

[0065] Step E50: After the last operator's forward computation is completed at the end edge computing node in the pipeline execution sequence, the final computation vector is used as the inference output of the corresponding inference micro-batch. The inference outputs of each inference micro-batch are then summarized to obtain the complete inference result.

[0066] Step S60 includes steps F10 to F50: Step F10: For the first edge computing node in the pipeline execution sequence, after completing the forward computation of the current operator, the generated computation vector is passed to the next edge computing node in the pipeline execution sequence through the ad hoc network, and the computation vector is cached locally as the output vector.

[0067] Step F20: For each edge computing node in the pipeline execution sequence except the first edge computing node, after receiving the computation vector transmitted by the previous edge computing node and completing the forward computation of the current operator, the generated computation vector is associated and encapsulated with the received computation vector to generate an authentication return packet, which is then transmitted back to the previous edge computing node through the ad hoc network.

[0068] Step F30: For each edge computing node, receive the authentication return packet sent back by the next edge computing node in the pipeline execution sequence, extract the received computation vector in the authentication return packet, compare it with the locally cached output vector for consistency, and generate the authentication result.

[0069] Step F40: For each edge computing node in the pipeline execution sequence except the first edge computing node, after receiving the authentication confirmation sent by the previous edge computing node, the locally generated computing vector is transmitted to the next edge computing node in the pipeline execution sequence through the ad hoc network, and the computing vector is cached locally as the output vector.

[0070] Step F50: If the last edge computing node in the pipeline execution sequence completes the forward computation of the last operator and the authentication results of each hop are consistent, the complete inference result is output as the final consensus result.

[0071] Furthermore, the method also includes steps G10 to G60: Step G10: If an abnormal computing power is detected in the target edge computing power node, the computing power fingerprint of the target edge computing power node is obtained as the fault computing power fingerprint, and the set of idle edge computing power nodes in the idle state is determined among the other edge computing power nodes in the ad hoc network.

[0072] Step G20: If the set of idle edge computing power nodes is not empty, calculate the fingerprint similarity between the computing power fingerprint of each idle edge computing power node and the fingerprint of the faulty computing power node, and select the idle edge computing power node with the highest fingerprint similarity as the takeover node.

[0073] Step G30: When the set of idle edge computing nodes is empty, calculate the fingerprint similarity between the computing power fingerprint of each edge computing node currently performing operator calculation and the fingerprint of the faulty computing node in the self-organizing network, and determine the takeover node.

[0074] Step G40: Send a takeover request to the takeover node. After completing the operator calculation it is currently executing, the takeover node releases its current computing resources and retrieves and loads the model parameter fragments corresponding to the target edge computing node from the local pre-stored model parameter library according to the takeover request.

[0075] Step G50: The takeover node takes over the execution position of the target edge computing node in the pipeline execution sequence, executes the operator calculation corresponding to the target edge computing node, and continues to pass the generated calculation vector to the next edge computing node according to the pipeline execution sequence.

[0076] Step G60: Perform fault isolation on the target edge computing node, remove it from the pipeline execution sequence, and update the computing node allocation scheme based on the takeover node.

[0077] Specifically, the method proposed in this embodiment revolves around the core issue of "how to enable visual language models to complete reliable inference in a distributed and collaborative manner in a completely off-grid heterogeneous edge environment". The overall technical chain can be summarized into seven stages: model structured decomposition, heterogeneous resource perception, optimal task allocation, on-demand parameter loading, pipeline collaborative execution, decentralized consensus verification, and automatic fault takeover.

[0078] In the model structured decomposition stage, the pre-trained visual language model is first decomposed layer by layer into indivisible atomic operators. Each atomic operator corresponds to an independent forward computation operation in the model, such as a matrix multiplication, a normalization operation, or an activation function operation. Subsequently, based on the data flow dependencies of each atomic operator in the forward inference process, a directed acyclic graph is constructed with atomic operators as vertices and data flow directions as directed edges, forming an initial global task dependency graph. On this basis, operator fusion is performed on adjacent atomic operators in the initial global task dependency graph that meet the fusion conditions. For example, continuous linear transformations and bias additions are merged into a single operator node, thereby reducing the number of vertices and cross-node communication times in the graph, resulting in the final global task dependency graph. This graph explicitly transforms the complete inference process, which was originally coupled within the model, into a set of independently schedulable discrete computation tasks, providing a structured descriptive foundation for subsequent resource awareness and task allocation.

[0079] In the heterogeneous resource awareness stage, the system sends probe requests to each edge computing node in the ad hoc network at a preset detection cycle to obtain the computing power status information returned by each node. This information covers dynamic indicators such as the node's current computing load, available memory, and energy reserves. Simultaneously, the system extracts the operator type and tensor shape of all operators corresponding to each vertex from the global task dependency graph. After deduplication, a representative operator set is formed, covering all computational modes involved in the model inference process. For each edge computing node, benchmark tests are performed on each representative operator in the representative operator set, recording the actual time consumption and resource usage when each node executes each representative operator, i.e., computing power usage characteristics. Finally, the computing power status information of each node is associated and encapsulated with the computing power usage characteristics of its corresponding representative operators to generate a computing power fingerprint for each edge computing node. The computing power fingerprint is essentially a structured and quantitative representation of the heterogeneous computing capabilities of each node, accurately reflecting the differences in actual execution efficiency of different hardware architectures and chip models for various types of operators, providing a precise measurement basis for subsequent task allocation.

[0080] In the optimal task allocation phase, the system performs a two-stage segmentation of the global task dependency graph. The first stage is initial segmentation: each directed edge in the global task dependency graph is assigned a communication cost weight, and each vertex is assigned a computational cost weight. The path with the largest cumulative computational cost weight is identified as the critical path. The graph is then divided equally along this critical path according to the cumulative computational cost weight, determining the initial set of splitting edges. Based on this, the global task dependency graph is segmented into multiple initial subgraphs, obtaining an initial segmentation scheme. The second stage is iterative refinement: the sum of the communication cost weights of all directed edges across subgraphs in the current segmentation scheme is calculated as the current segmentation cost. All vertices located at the boundaries of the subgraphs are traversed, and the change in segmentation cost after migrating each boundary vertex to its adjacent subgraph is evaluated. The boundary vertex that maximizes the reduction in segmentation cost is selected for migration, and the sum of the computational cost weights of all subgraphs after migration must satisfy a preset equilibrium constraint. This iteration is repeated until there are no longer any valid migration operations that reduce the segmentation cost and satisfy the equilibrium constraint. The resulting multiple subgraphs are the lightweight task operator dependency subgraphs. This two-stage strategy balances the initial rationality of the partitioning with the continuous optimization of communication costs, ensuring that each subgraph is balanced in terms of computational scale and minimizes cross-subgraph communication.

[0081] After obtaining the lightweight task operator-dependent subgraphs, the system further determines the computational complexity characteristics and cross-subgraph communication characteristics of each subgraph. The computational complexity characteristics represent the cumulative computational cost of each atomic operator within the subgraph, while the cross-subgraph communication characteristics represent the data transfer cost between the subgraph and its neighboring subgraphs. The computational complexity characteristics of each subgraph are matched with the computing power fingerprints of each edge computing node to determine the fit between each subgraph and each node. Simultaneously, based on the cross-subgraph communication characteristics of each subgraph and the link communication capabilities between nodes in the ad hoc network, the inter-subgraph communication cost matrix is ​​determined. Finally, with minimizing global inference latency as the optimization objective, the computing power node allocation scheme is determined by comprehensively calculating the fit and the communication cost matrix, i.e., the mapping relationship between each lightweight task operator-dependent subgraph and each edge computing node.

[0082] In the on-demand parameter loading stage, the system determines the operator allocation list corresponding to each edge computing node based on the computing power node allocation scheme, generates operator allocation requests accordingly, and sends them to the corresponding nodes through the ad hoc network. Upon receiving the operator allocation request, each edge computing node retrieves and loads the corresponding model parameter fragments from its locally pre-stored model parameter library based on the parameter addressing information of each atomic operator in the request, rather than loading the complete model parameters. This achieves distributed storage and on-demand loading of model parameters within the ad hoc network, significantly reducing the storage footprint of each node. After loading is complete, each node performs parameter integrity verification on the loaded model parameter fragments. If the verification results from all nodes indicate that the verification passed, each node returns to the loading-ready state, triggering the initiation of subsequent collaborative inference.

[0083] In the pipelined collaborative execution phase, the system determines the pipelined execution sequence based on the mapping relationship between the lightweight task operator dependency subgraph and each edge computing node in the computing node allocation scheme, as well as the data flow dependency order between the subgraphs. The input data to be inferred is divided into multiple inference micro-batches according to a preset micro-batch granularity. Each inference micro-batch independently completes inference through the pipelined execution sequence. Each edge computing node executes the forward computation of the assigned operator sequentially according to the execution timing in the pipelined execution sequence, generates the computation vector corresponding to the current operator, and after completing the computation of the current operator, passes the computation vector to the next edge computing node in the pipeline through the ad hoc network, triggering the next node to execute the forward computation of the subsequent operator. When the edge computing node at the end of the pipeline completes the forward computation of the last operator, it uses the final computation vector as the inference output of the corresponding inference micro-batch. Summarizing the inference outputs of all inference micro-batches yields the complete inference result. The pipelined mechanism allows different micro-batches to be in computation state simultaneously on different nodes, enabling parallel utilization of the computing resources of each node and effectively shortening the overall completion time of multi-batch inference.

[0084] In the decentralized consensus verification phase, the first edge computing node in the pipeline execution sequence, after completing the forward computation of the current operator, passes the generated computation vector to the next node and caches the computation vector locally as an output vector. Each edge computing node in the pipeline, excluding the first node, receives the computation vector passed from the previous node, completes the forward computation of its current operator, associates and encapsulates the generated computation vector with the received computation vector, generates an authentication return packet, and sends it back to the previous node via the ad hoc network. Upon receiving the authentication return packet from the next node, each node extracts the computation vector carried within, performs a consistency comparison with the locally cached output vector, and generates an authentication result. Each node can only continue to pass its locally generated computation vector downstream after receiving authentication confirmation from the previous node. When the last node in the pipeline completes the forward computation of the last operator and all hop authentication results are consistent, the complete inference result is output as the final consensus result. In an off-network environment lacking a trusted central node, this mechanism ensures the correctness of inference results through hop-by-hop autonomous verification between adjacent nodes, avoiding the silent transmission of erroneous results due to single-point calculation errors or communication interference.

[0085] In the automatic failover phase, the system continuously monitors the operational status of each edge computing node. When an abnormal computing power is detected in a target edge computing node, its computing power fingerprint is obtained as the faulty computing power fingerprint. The system then identifies a set of currently idle edge computing nodes among the remaining nodes in the ad hoc network. If the set of idle nodes is not empty, the fingerprint similarity between the computing power fingerprint of each idle node and the faulty computing power fingerprint is calculated, and the idle node with the highest similarity is selected as the takeover node. If the set of idle nodes is empty, the takeover node is determined from among the nodes currently executing operator computations using the same fingerprint similarity criterion. After completing its currently executing operator computation, the takeover node releases its current computing power resources, retrieves and loads the model parameter fragments corresponding to the target edge computing node from its local model parameter library according to the takeover request, and then takes over the execution position of the target node in the pipeline execution sequence, continuing to execute the corresponding operator computation and passing the generated computation vector to the next node in the pipeline. At the same time, the system performs fault isolation on the target edge computing nodes, removes them from the pipeline execution sequence, and updates the computing node allocation scheme based on the takeover node, thereby achieving seamless replacement of faulty nodes without interrupting the overall inference process and ensuring the system's continuous service capability in harsh off-network environments.

[0086] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the lightweight edge collaborative reasoning method for high-risk off-grid multimodal large models in this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0087] This application also provides a lightweight edge collaborative inference system for high-risk off-grid multimodal large-scale models. Please refer to [link / reference]. Figure 2 The high-risk off-grid multimodal large-scale lightweight edge collaborative inference system includes: The graph construction module performs operator-level decomposition on the pre-trained visual language model to construct a global task dependency graph.

[0088] The computing power determination module is used to periodically detect the computing power pool of edge nodes, determine the effective computing power of each edge computing power node in the ad hoc network, and determine the computing power fingerprint of each edge computing power node by combining the global task dependency graph.

[0089] The task allocation module is used to decompose the global task dependency graph to obtain a lightweight task operator dependency subgraph, and combine the computing power fingerprints of each edge computing power node to determine the computing power node allocation scheme.

[0090] The parameter loading module is used to send operator allocation requests to each edge computing node, so that each edge computing node can load the necessary model parameters according to the operator allocation request.

[0091] The collaborative reasoning module is used to initiate edge collaborative reasoning, which uses a pipeline approach to execute operator calculations sequentially through each edge computing node.

[0092] The result output module is used for each edge computing node to pass computation vectors through a self-organizing network, verify the computation results with each other through a decentralized consensus algorithm, and output the final consensus result.

[0093] The high-risk off-grid multimodal large-scale lightweight edge collaborative inference system provided in this application adopts the high-risk off-grid multimodal large-scale lightweight edge collaborative inference method in the above embodiments, which can solve the technical problems in related technologies. Compared with related technologies, the beneficial effects of the high-risk off-grid multimodal large-scale lightweight edge collaborative inference system provided in this application are the same as the beneficial effects of the high-risk off-grid multimodal large-scale lightweight edge collaborative inference method provided in the above embodiments, and other technical features of the high-risk off-grid multimodal large-scale lightweight edge collaborative inference system are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0094] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A lightweight edge collaborative inference method for high-risk off-grid multimodal large models, characterized in that, The method includes: The pre-trained visual language model is decomposed at the operator level to construct a global task dependency graph; Periodically check the edge node computing power pool to determine the effective computing power of each edge computing power node in the self-organizing network, and combine it with the global task dependency graph to determine the computing power fingerprint of each edge computing power node; The global task dependency graph is decomposed to obtain a lightweight task operator dependency subgraph. Combined with the computing power fingerprint of each edge computing power node, the computing power node allocation scheme is determined. Send operator allocation requests to each edge computing node, so that each edge computing node can load the necessary model parameters according to the operator allocation requests; Initiate edge collaborative reasoning and use a pipeline approach to sequentially execute operator calculations through each edge computing node; Each edge computing node transmits computing vectors through a self-organizing network, verifies the computing results with each other through a decentralized consensus algorithm, and outputs the final consensus result.

2. The lightweight edge collaborative inference method for high-risk off-grid multimodal large models according to claim 1, characterized in that, The steps of decomposing the pre-trained visual language model at the operator level and constructing a global task dependency graph include: The pre-trained visual language model is decomposed layer by layer into indivisible atomic operators; Based on the data flow dependencies of each atomic operator in the forward inference process, a directed acyclic graph is constructed as the initial global task dependency graph; wherein, the vertices of the directed acyclic graph correspond to atomic operators, and the directed edges represent the outputs of the upstream operators constituting the inputs of the downstream operators; Operator fusion is performed on adjacent atomic operators that satisfy the fusion condition in the initial global task dependency graph to obtain the global task dependency graph.

3. The lightweight edge collaborative inference method for high-risk off-grid multimodal large models according to claim 2, characterized in that, The steps of periodically detecting the edge node computing power pool, determining the effective computing power of each edge computing power node in the ad hoc network, and determining the computing power fingerprint of each edge computing power node by combining the global task dependency graph include: Send probe requests to each edge computing node in the self-organizing network at a preset detection cycle to obtain the computing power status information returned by each edge computing node; Based on the operator type and tensor shape of the operators corresponding to each vertex in the global task dependency graph, a representative set of operators is determined; the representative set of operators is the set of operators obtained after deduplicating the operator types and tensor shapes in the global task dependency graph. For each edge computing node, benchmark tests of each representative operator in the representative operator set are executed one by one, and the computing power consumption characteristics of each edge computing node executing each representative operator are recorded. The computing power status characteristics of each edge computing node are associated and encapsulated with the computing power occupancy characteristics of its corresponding representative operators to generate the computing power fingerprint of each edge computing node.

4. The lightweight edge collaborative inference method for high-risk off-grid multimodal large models as described in claim 3, characterized in that, The steps of decomposing the global task dependency graph to obtain a lightweight task operator dependency subgraph, and combining the computing power fingerprints of each edge computing power node to determine the computing power node allocation scheme include: The global task dependency graph is divided according to a preset decomposition strategy to obtain multiple lightweight task operator dependency subgraphs; The computational cost characteristics and cross-subgraph communication characteristics of each lightweight task operator dependent subgraph are determined; the computational cost characteristics represent the cumulative computational cost of each atomic operator within the lightweight task operator dependent subgraph, and the cross-subgraph communication characteristics represent the data transfer cost between the lightweight task operator dependent subgraph and adjacent lightweight task operator dependent subgraphs. The computational characteristics of each lightweight task operator dependency subgraph are matched with the computational fingerprint of each edge computing node to determine the fit between each lightweight task operator dependency subgraph and each edge computing node. Based on the cross-subgraph communication volume characteristics of each lightweight task operator-dependent subgraph and the link communication capabilities between each edge computing node in the ad hoc network, the inter-subgraph communication cost matrix is ​​determined. Based on the computational fit between each lightweight task operator dependency subgraph and each edge computing node, as well as the communication cost matrix between the subgraphs, a computing node allocation scheme is determined with the goal of minimizing global inference latency.

5. The lightweight edge collaborative inference method for high-risk off-grid multimodal large models according to claim 4, characterized in that, The step of segmenting the global task dependency graph according to a preset decomposition strategy to obtain multiple lightweight task operator dependency subgraphs includes: Assign a communication cost weight to each directed edge in the global task dependency graph, and an computation cost weight to each vertex; The path with the largest cumulative computational cost weight in the global task dependency graph is identified as the critical path. The critical path is then divided equally according to the cumulative computational cost weight to determine the initial set of splitting edges. Based on the initial set of splitting edges, the global task dependency graph is divided into multiple initial subgraphs to obtain the initial splitting scheme. Perform an iterative refinement operation on the initial segmentation scheme, and calculate the sum of the communication cost weights of all directed edges across the subgraph in the current segmentation scheme as the current segmentation cost; Traverse all vertices located on the boundaries of the subgraph and calculate the change in segmentation cost after migrating each boundary vertex to its adjacent subgraph. Select the boundary vertices that maximize the reduction in segmentation cost to perform migration, and ensure that the sum of the computational cost weights of each subgraph after migration satisfies the preset equilibrium constraint. Repeat the iterative refinement operation until there is no legal migration operation that reduces the cutting cost and satisfies the preset equilibrium constraint. The resulting initial subgraphs are then used as the dependent subgraphs of the multiple lightweight task operators.

6. The lightweight edge collaborative inference method for high-risk off-grid multimodal large models according to claim 5, characterized in that, The step of sending operator allocation requests to each edge computing node, so that each edge computing node loads the necessary model parameters according to the operator allocation requests, includes: Based on the computing power node allocation scheme, determine the operator allocation list corresponding to each edge computing power node; Based on the operator allocation list of each edge computing node, generate corresponding operator allocation requests, and send each operator allocation request to the corresponding edge computing node through the self-organizing network; For each edge computing node, based on the parameter addressing information of each atomic operator in the received operator allocation request, the model parameter fragments of the corresponding atomic operator are retrieved from the locally pre-stored model parameter library and loaded. For each edge computing node, perform parameter integrity verification on each loaded model parameter segment and generate verification results; If the verification results of each edge computing node indicate that the verification has passed, each edge computing node returns to the loading ready state to trigger the step of starting edge collaborative reasoning.

7. The lightweight edge collaborative inference method for high-risk off-grid multimodal large models according to claim 6, characterized in that, The steps for initiating edge collaborative inference, which employs a pipelined approach to sequentially execute operator calculations through each edge computing node, include: Based on the mapping relationship between each lightweight task operator dependency subgraph and each edge computing node in the computing power node allocation scheme, and the data flow dependency order between each lightweight task operator dependency subgraph, the pipeline execution sequence is determined. The input data to be inferred is divided into multiple inference micro-batches according to a preset micro-batch granularity; each inference micro-batch independently completes inference through the pipeline execution sequence. For each edge computing node, the forward computation of each assigned operator is executed sequentially according to the execution timing in the pipeline execution sequence to generate the computation vector corresponding to the current operator. After completing the forward computation of the current atomic operator, each edge computing node transmits the computation vector to the next edge computing node in the pipeline execution sequence through the ad hoc network, so as to trigger the next edge computing node to execute the forward computation of subsequent operators. After the last edge computing node in the pipeline execution sequence completes the forward computation of the last operator, the final computation vector is used as the inference output of the corresponding inference micro-batch, and the inference outputs of each inference micro-batch are summarized to obtain the complete inference result.

8. The lightweight edge collaborative inference method for high-risk off-grid multimodal large models according to claim 7, characterized in that, The steps for each edge computing node to pass computation vectors through a self-organizing network, verify computation results with each other using a decentralized consensus algorithm, and output the final consensus result include: For the first edge computing node in the pipeline execution sequence, after completing the forward computation of the current operator, the generated computation vector is passed to the next edge computing node in the pipeline execution sequence through the ad hoc network, and the computation vector is cached locally as an output vector. For each edge computing node in the pipeline execution sequence except the first edge computing node, after receiving the computing vector transmitted by the previous edge computing node and completing the forward calculation of the current operator, the generated computing vector is associated and encapsulated with the received computing vector to generate an authentication return packet, and the authentication return packet is returned to the previous edge computing node through the self-organizing network. For each edge computing node, the authentication return packet sent back by the next edge computing node in the pipeline execution sequence is received, the received computation vector in the authentication return packet is extracted, and the consistency is compared with the locally cached output vector to generate an authentication result. For each edge computing node in the pipeline execution sequence except the first edge computing node, after receiving the authentication confirmation sent by the previous edge computing node, the locally generated computing vector is transmitted to the next edge computing node in the pipeline execution sequence through the ad hoc network, and the computing vector is cached locally as an output vector. If the last edge computing node in the pipeline execution sequence completes the forward computation of the last operator and the authentication results of each hop are consistent, the complete inference result will be output as the final consensus result.

9. The lightweight edge collaborative inference method for high-risk off-grid multimodal large models as described in claim 1, characterized in that, The method further includes: If an abnormal computing power is detected in the target edge computing power node, the computing power fingerprint of the target edge computing power node is obtained as the fault computing power fingerprint, and the set of idle edge computing power nodes in the idle state is determined among the other edge computing power nodes in the ad hoc network. When the set of idle edge computing power nodes is not empty, calculate the fingerprint similarity between the computing power fingerprint of each idle edge computing power node and the fingerprint of the faulty computing power, and select the idle edge computing power node with the highest fingerprint similarity as the takeover node. When the set of idle edge computing nodes is empty, among the edge computing nodes currently performing operator calculations in the ad hoc network, the fingerprint similarity between the computing power fingerprint of each edge computing node currently performing operator calculations and the fingerprint of the faulty computing power is calculated to determine the takeover node. A takeover request is sent to the takeover node. After completing the operator calculation it is currently executing, the takeover node releases its current computing resources and retrieves and loads the model parameter fragments corresponding to the target edge computing node from the locally pre-stored model parameter library according to the takeover request. The takeover node takes over the execution position of the target edge computing node in the pipeline execution sequence, performs the operator calculation corresponding to the target edge computing node, and continues to pass the generated calculation vector to the next edge computing node according to the pipeline execution sequence; Fault isolation is performed on the target edge computing node, removing it from the pipeline execution sequence, and the computing node allocation scheme is updated based on the takeover node.

10. A lightweight edge collaborative inference system for high-risk off-grid multimodal large models, characterized in that, The system employs the lightweight edge collaborative inference method for high-risk off-grid multimodal large models as described in any one of claims 1 to 9, wherein the system comprises: The graph construction module performs operator-level decomposition on the pre-trained visual language model to construct a global task dependency graph; The computing power determination module is used to periodically detect the computing power pool of edge nodes, determine the effective computing power of each edge computing power node in the ad hoc network, and determine the computing power fingerprint of each edge computing power node by combining the global task dependency graph. The task allocation module is used to decompose the global task dependency graph to obtain a lightweight task operator dependency subgraph, and combine the computing power fingerprint of each edge computing power node to determine the computing power node allocation scheme. The parameter loading module is used to send operator allocation requests to each edge computing node, so that each edge computing node can load the necessary model parameters according to the operator allocation request. The collaborative reasoning module is used to initiate edge collaborative reasoning, which uses a pipeline approach to execute operator calculations sequentially through each edge computing node. The result output module is used for each edge computing node to pass computation vectors through a self-organizing network, verify the computation results with each other through a decentralized consensus algorithm, and output the final consensus result.

Images