Cloud edge-end collaborative deployment method and system of image reasoning multi-mode neural network

Abstract

The invention provides a cloud edge-end collaborative deployment method and system for an image reasoning multi-mode neural network, and is applied to the field of artificial intelligence, and the method comprises the steps: abstracting a preset multi-mode neural network, and obtaining an original calculation graph; based on the calculation node set and the data dependence edge set, splitting the original calculation graph to obtain a plurality of connected sub-graphs, the categories of the plurality of connected sub-graphs including a single-modal processing sub-graph, a cross-modal interaction sub-graph and a task specific sub-graph; based on the topological sequence of the plurality of connected sub-graphs, determining segmentation points of each connected sub-graph; dividing the plurality of connected sub-graphs according to the segmentation points to obtain a plurality of segmentation parts; and integrally dividing the multi-modal neural network based on the plurality of segmentation parts according to a genetic algorithm, and respectively deploying the integrally divided multi-modal neural network to a terminal device, an edge server and a cloud platform. According to the invention, a flexible and efficient model segmentation and deployment mechanism can be realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a cloud-edge-device collaborative deployment method and system for image reasoning multimodal neural networks. Background Technology

[0002] To enhance environmental perception and task response capabilities, deep neural network (DNN) technology has been widely applied in multimodal processing workflows such as power equipment identification, alarm classification, and speech semantic understanding. However, multimodal DNN models typically have characteristics such as complex structure, high computational intensity, and massive data volume, which pose significant deployment challenges for augmented reality (AR) terminal devices.

[0003] The "cloud-edge-device" collaborative reasoning architecture proposed in recent years is considered a feasible solution for multimodal reasoning in power operation and maintenance scenarios. Its basic idea is to divide the deep learning model according to its structure and deploy it on AR terminals, edge servers and cloud platforms for collaborative execution, so as to achieve computational load balancing and communication cost control throughout the entire reasoning process.

[0004] However, several challenges remain in actual deployment: for example, how to dynamically partition the model structure on different device layers to balance latency and energy consumption; and how to handle intermodal alignment and data dependencies to ensure semantic consistency of multimodal inputs in heterogeneous systems. Summary of the Invention

[0005] This invention provides a cloud-edge-device collaborative deployment method and system for image inference multimodal neural networks, which solves the problem that the existing technology of dynamically partitioning model structures on different device layers cannot meet the dual requirements of real-time performance and energy efficiency, and realizes a flexible and efficient model segmentation and deployment mechanism.

[0006] This invention provides a cloud-edge-device collaborative deployment method for image inference multimodal neural networks, comprising the following steps: Abstracting a preset multimodal neural network to obtain an original computation graph, wherein the original computation graph is a directed acyclic graph, and the original computation graph includes: a set of computation nodes and a set of data-dependent edges; splitting the original computation graph based on the set of computation nodes and the set of data-dependent edges to obtain multiple connected subgraphs, wherein the multiple connected subgraphs include: single-modal processing subgraphs, cross-modal interaction subgraphs, and task-specific subgraphs; determining the splitting point of each connected subgraph based on the topological order of the multiple connected subgraphs; dividing the multiple connected subgraphs according to the splitting point to obtain multiple segmented parts; performing an overall partitioning of the multimodal neural network based on the multiple segmented parts using a genetic algorithm, and deploying the overall partitioned multimodal neural network on terminal devices, edge servers, and cloud platforms respectively.

[0007] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network includes the following steps: First, based on the set of computing nodes and the set of data-dependent edges, the original computation graph is split to obtain multiple connected subgraphs. This includes: determining the data connection relationships and semantic functions of each node based on the set of computing nodes and the set of data-dependent edges; second, based on the data connection relationships and semantic functions, identifying and dividing the substructures located at the front end of the multimodal neural network that only receive a single modal input stream to obtain a single-modal processing subgraph; third, based on the data connection relationships, locating and dividing the data fusion substructures in the multimodal neural network that simultaneously receive at least two modal inputs to obtain a cross-modal interaction subgraph; and fourth, based on the semantic functions, identifying and dividing the substructures located downstream of the multimodal neural network and used for decision prediction or generation tasks to obtain task-specific subgraphs.

[0008] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network is provided, wherein the multimodal neural network includes... There are several levels, and the dividing points are respectively the first dividing point. With the second dividing point The step of dividing the multiple connected subgraphs according to the cutting point to obtain multiple segmented parts includes: according to the first cutting point With the second dividing point For the multiple connected subgraphs Divide into parts to obtain the first part. Part Two and Part Three The first part is deployed on a terminal device, the second part is deployed on an edge server, and the third part is deployed on a cloud platform.

[0009] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network is provided. The method further includes: determining the overall inference delay of the first part, the second part, and the third part based on the first segmentation point and the second segmentation point, wherein the method includes: in, This indicates the first dividing point. Indicates the second dividing point. This indicates the overall reasoning delay. Indicates the overall computational delay. This indicates the overall communication delay.

[0010] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network is provided. The method further includes: determining the overall computational latency based on the computing power of the terminal device, the computing power of the edge server, and the computing power of the cloud platform, wherein the method includes: in, This indicates the first dividing point. Indicates the second dividing point. This indicates the overall computational delay. Indicates the first Hierarchical computational load, This indicates the computing power of the terminal device. This indicates the computing power of the edge server. This indicates the computing power of the cloud platform.

[0011] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network is provided. The method further includes: determining the overall communication latency based on the output tensor size and communication bandwidth between the terminal device, the edge server, and the cloud platform, wherein the method includes: in, This indicates the overall communication delay. This indicates that the multimodal neural network is at the first split point. The size of the output feature tensor after segmentation. This indicates that the multimodal neural network is at the second split point. The size of the output feature tensor after segmentation. This represents the communication bandwidth from the terminal device to the edge server. This indicates the communication bandwidth from the edge server to the cloud platform.

[0012] This invention also provides a cloud-edge-device collaborative deployment system for image inference multimodal neural networks, comprising the following modules: an abstraction module, used to abstract a preset multimodal neural network to obtain an original computation graph, wherein the original computation graph is a directed acyclic graph, and the original computation graph includes: a set of computation nodes and a set of data-dependent edges; a splitting module, used to split the original computation graph based on the set of computation nodes and the set of data-dependent edges to obtain multiple connected subgraphs, wherein the multiple connected subgraphs include: a single-modal processing subgraph, a cross-modal interaction subgraph, and a task-specific subgraph; a determination module, used to determine the splitting point of each connected subgraph based on the topological order of the multiple connected subgraphs; a partitioning module, used to partition the multiple connected subgraphs according to the splitting point to obtain multiple partitioned parts; and a deployment module, used to perform overall partitioning of the multimodal neural network based on the multiple partitioned parts using a genetic algorithm, and deploy the overall partitioned multimodal neural network on terminal devices, edge servers, and cloud platforms respectively.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cloud-edge-device collaborative deployment method of the image inference multimodal neural network as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cloud-edge-device collaborative deployment method of the image inference multimodal neural network as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cloud-edge-device collaborative deployment method of the image inference multimodal neural network as described above.

[0016] The present invention provides a cloud-edge-device collaborative deployment method and system for image inference multimodal neural networks. First, it abstracts the multimodal neural network into a directed acyclic graph (including computational nodes and data-dependent edges) and accurately models the model structure and data flow. Second, based on node connection relationships and semantic functions, it decomposes the original computational graph into three types of functionalized connected subgraphs: single-modal processing subgraphs (front-end single-modal input), cross-modal interaction subgraphs (mid-section multimodal fusion), and task-specific subgraphs (back-end decision output). This step provides targeted structural units for deployment through semantic classification. Then, it determines the internal interruptible split points according to the subgraph topological order to ensure data flow continuity. After that, it divides each subgraph into continuous segments according to the split points. Finally, based on the functional characteristics of the subgraphs, it deploys single-modal segments on terminal devices (near-source processing of original input), cross-modal segments on edge servers (performing modal fusion), and task-specific segments on cloud platforms (completing complex decisions), forming an edge-cloud collaborative inference scheme. By using function-driven structured partitioning and topological constraint segmentation, the deployment complexity of multimodal models is significantly reduced. At the same time, heterogeneous resource allocation (light computing at the terminal, medium integration at the edge, and heavy tasks in the cloud) naturally balances inference real-time performance and energy consumption, thereby optimizing overall computing efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the cloud-edge-device collaborative deployment method for image inference multimodal neural networks provided by the present invention.

[0019] Figure 2 This is a schematic diagram of DAG segmentation based on subgraphs provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the multimodal model deployment architecture based on cloud-edge-device collaboration provided by the present invention.

[0021] Figure 4 This is a flowchart of the genetic algorithm for model segmentation deployment provided by the present invention.

[0022] Figure 5 This is a comparison of latency for different deployment strategies provided by this invention.

[0023] Figure 6 This invention provides a comparison of the energy consumption of terminal devices under different deployment strategies.

[0024] Figure 7 This is a comparison of the adaptability of different deployment strategies provided by this invention.

[0025] Figure 8 This refers to the relative depth of the segmentation position under different terminal device computing capabilities provided by the present invention.

[0026] Figure 9 This is a schematic diagram of the cloud-edge-device collaborative deployment system for image inference multimodal neural networks provided by the present invention.

[0027] Figure 10 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] Currently, power operation and maintenance systems are gradually developing towards intelligence and remote operation. With the rapid development of augmented reality (AR), wearable terminals, and multimodal sensing technologies, AR-based power operation and maintenance is becoming an important part of the new generation of work modes. By deploying lightweight terminals (such as AR glasses, smart helmets, and handheld terminals) with visual sensors, voice input, and human-computer interaction capabilities, on-site maintenance personnel can acquire multi-source information such as equipment images, voice descriptions, and environmental status, enabling complex tasks such as remote assisted diagnosis, intelligent navigation guidance, and efficient fault handling. In such scenarios, low-latency, low-power image recognition and multimodal information reasoning capabilities become crucial to ensuring the stable operation of the system.

[0030] To enhance environmental perception and task response capabilities, deep neural network (DNN) technology has been widely applied in multimodal processing workflows such as power equipment identification, alarm classification, and speech semantic understanding. However, multimodal DNN models typically possess characteristics such as complex structure, high computational intensity, and massive data volume, posing significant deployment challenges for AR terminal devices. Current inference strategies mainly fall into two categories: one is executing the complete model locally on the terminal, which ensures on-site data processing and privacy protection but faces resource bottlenecks such as insufficient computing power and excessive energy consumption; the other is sending the entire task to the cloud for execution, which offers higher resource utilization but suffers from problems such as high transmission latency, high bandwidth requirements, unstable connections, and privacy leaks, making it particularly difficult to adapt to outdoor emergency repairs or special inspection environments. Therefore, how to balance AR terminal performance and system responsiveness while meeting the needs of real-time multimodal perception has become a critical issue that urgently needs to be addressed.

[0031] Therefore, the "cloud-edge-device" collaborative reasoning architecture proposed in recent years is considered a feasible solution for multimodal reasoning in power operation and maintenance scenarios. Its basic idea is to segment the deep learning model according to its structure and deploy it collaboratively across AR terminals, edge servers, and the cloud platform to achieve computational load balancing and communication cost control throughout the inference process. However, several challenges remain in actual deployment: for example, how to dynamically partition the model structure at different device layers to balance latency and energy consumption; how to handle intermodal alignment and data dependencies to ensure semantic consistency of multimodal inputs in heterogeneous systems. To address these issues, a structurally complete, performance-controllable, and flexible efficient reasoning optimization mechanism is still lacking.

[0032] To at least address some of the aforementioned problems, this invention provides a cloud-edge-device collaborative deployment method for image inference multimodal neural networks.

[0033] In related technologies, image recognition and fault diagnosis are performed via remote servers. The advantage of this system lies in its ability to collect real-time on-site data through AR glasses, assisting maintenance personnel in identifying equipment faults and obtaining expert guidance. However, the system relies entirely on remote servers for image recognition, which can lead to delays in fault diagnosis and impact maintenance efficiency in situations with unstable networks or high latency. Furthermore, the system lacks edge computing support, preventing it from utilizing the local computing power of terminal devices for real-time inference, resulting in data transmission delays and computational bottlenecks. This problem is particularly prominent in maintenance tasks with high real-time requirements, failing to meet the demands for efficient management of response time and computing resources.

[0034] In this related technology, a design combining AR wearable devices with human body monitoring devices was introduced to focus on the safety and health of on-site operators, and to provide real-time power equipment status support through a central control host. While this solution combines health monitoring and operation and maintenance guidance, it also suffers from the problem of centralized system processing. All data relies on the central control host for calculation and decision-making, resulting in poor system scalability and flexibility. In practical applications, especially in resource-constrained or high-real-time-requirement environments, this centralized processing approach exacerbates computational bottlenecks and fails to efficiently allocate tasks to edge or terminal devices. Furthermore, the system does not consider the heterogeneous deployment and dynamic scheduling of deep learning models, and lacks intelligent collaboration and resource management between different computing nodes, impacting the overall system performance.

[0035] In related technologies, edge servers and algorithm servers are used to process image data, constructing a multi-point perception and collaborative framework. While this approach has advantages in terms of device diversity and image data acquisition, it still suffers from significant limitations. First, the system's image processing and anomaly detection rely on a pre-defined rule engine, lacking a flexible deep neural network model for more complex reasoning and diagnosis. Especially in multimodal data fusion and dynamic reasoning tasks in complex environments, this approach struggles to handle the coordination and real-time processing requirements between different data sources. Second, the system relies on substantial data transmission, particularly cross-device image data transmission, leading to high bandwidth consumption and potential transmission latency. This design cannot meet real-time response requirements, especially under conditions of limited bandwidth or unstable network conditions.

[0036] This invention aims to address the problems existing in multimodal deep neural network inference in current power operation and maintenance scenarios, such as prominent computational bottlenecks, high inference latency, and high energy consumption of terminal devices, which fail to meet the dual requirements of real-time performance and energy efficiency. To this end, this invention proposes an optimization method for a cloud-edge-device collaborative computing framework. Through intelligent task partitioning and efficient deployment, it enables collaborative inference among AR terminals, edge servers, and cloud resources, thereby reducing AR terminal energy consumption, optimizing system response time, and improving overall resource utilization and operation and maintenance efficiency. Ultimately, it meets the intelligent and reliable power operation and maintenance needs in resource-constrained environments.

[0037] To achieve the goals of optimizing inference latency and saving energy consumption of AR terminals, this invention proposes a multimodal neural network inference optimization scheme for heterogeneous resources, based on a cloud-edge-device collaborative computing architecture. The core of this scheme lies in designing a flexible and efficient model segmentation and deployment mechanism to meet the requirements of power operation and maintenance sites for inference real-time performance and terminal energy consumption control.

[0038] Optionally, the cloud-edge-device collaborative deployment method for image inference multimodal neural networks in this embodiment can be executed by a server, by a terminal device, or by both a server and a terminal device. Taking the execution of the cloud-edge-device collaborative deployment method for image inference multimodal neural networks in this embodiment by a server as an example.

[0039] Figure 1 This is a flowchart illustrating the cloud-edge-device collaborative deployment method for image inference multimodal neural networks provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps: Step 101: Abstract the preset multimodal neural network to obtain the original computation graph.

[0040] The original computation graph is a directed acyclic graph, which includes a set of computation nodes and a set of data dependency edges. To rationally express and analyze the model structure, this invention employs a Directed Acyclic Graph (DAG) as a unified structural modeling tool to represent the computational logic and information flow of the model. Specifically, a given multimodal neural network is abstracted into a computational graph. , where the set of nodes The set of edges represents the basic computational operations in the model (such as convolutional layers, attention layers, normalization layers, etc.). Indicates from node Output flow to node Data dependencies.

[0041] Compared to traditional unimodal networks, multimodal neural networks exhibit more complex and nonlinear structural features in their Directed Acyclic Graph (DAG) structures. Multimodal models often contain multiple parallel encoding paths, multiple branches such as modality alignment modules, and complex data convergence logic with tight dependencies. Therefore, traditional linear sequence-based partitioning methods are ill-suited to these nonlinear and functionally heterogeneous structural characteristics. To effectively reduce the difficulties caused by model structure in deployment optimization, and to better utilize the semantic structure and parallelism within the model, this invention proposes a structured partitioning method based on functional subgraphs.

[0042] Step 102: Based on the set of computational nodes and the set of data-dependent edges, the original computational graph is split to obtain multiple connected subgraphs.

[0043] Among them, the categories of multiple connected subgraphs include: single-modal processing subgraphs, cross-modal interaction subgraphs, and task-specific subgraphs; In this embodiment of the invention, the original computation graph is split based on the set of computation nodes and the set of data dependency edges to obtain multiple connected subgraphs of the original computation graph. It is divided into a set of semantically functionally consistent and structurally tightly coupled subgraphs. Each subgraph It is a computational graph A connected subgraph that satisfies and Common examples of subgraph types include: unimodal processing subgraphs, cross-modal interaction subgraphs, and task-specific subgraphs.

[0044] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network is provided. Based on the set of computing nodes and the set of data-dependent edges, the original computation graph is split to obtain multiple connected subgraphs, including: Based on the set of computing nodes and the set of data dependency edges, the data connection relationship and semantic function of each node are determined; Based on data connectivity and semantic functions, the substructure located at the front end of the multimodal neural network that only receives a single modal input stream is identified and divided to obtain a single-modal processing subgraph; Based on data connectivity, the data fusion substructure in a multimodal neural network that simultaneously receives input from at least two modalities is located and partitioned to obtain a cross-modal interaction subgraph. Based on semantic functions, substructures located downstream of multimodal neural networks and used for decision prediction or generation tasks are identified and divided to obtain task-specific subgraphs.

[0045] refer to Figure 2 , Figure 2 This is a schematic diagram of DAG segmentation based on subgraphs provided by the present invention. Figure 2 This paper presents an example of representing a multimodal model as a DAG and dividing it into different subgraphs to achieve segmentation. By representing the model as a DAG and splitting it into different subgraphs, the structure of the multimodal model is more clearly expressed, which helps to capture the diverse logical relationships within the multimodal model and reduces the complexity of deployment optimization problems.

[0046] In this embodiment of the invention, during actual partitioning, the model is structurally deconstructed based on the original computation graph (directed acyclic graph) of the multimodal neural network, combined with the connection relationships and semantic functions between nodes. Within the entire network structure, using the namespace of nodes and upstream information flow as clues, the substructures located at the front end corresponding to a single modal input stream are first identified and marked as single-modal processing subgraphs. Next, the regions where cross-modal fusion operations occur—that is, data fusion modules (substructures) that simultaneously receive inputs from two or more modalities—are located and defined as cross-modal interaction subgraphs. Finally, the downstream output subgraphs used for decision prediction or generation tasks are identified and labeled as task-specific subgraphs.

[0047] Through the embodiments of the present invention, based on the node data connection relationship and semantic function, the original computation graph is split into a single-modal processing subgraph (lightweight front-end), a cross-modal interaction subgraph (centralized processing in the fusion layer), and a task-specific subgraph (customized back-end), matching different hardware resources (such as single-modal processing at the edge and complex fusion execution in the cloud).

[0048] Step 103: Determine the splitting point of each connected subgraph based on the topological order of the multiple connected subgraphs.

[0049] In this embodiment of the invention, after completing the functional subgraph partitioning of the model structure, possible splitting positions are further selected within each subgraph, and possible splitting points are defined for each subgraph, thereby enabling the entire model to be split and deployed to terminal devices, edge servers, and cloud platforms.

[0050] Step 104: Divide the multiple connected subgraphs according to the splitting points to obtain multiple split parts.

[0051] In this embodiment of the invention, the partitioning result of each subgraph provides a clear structural basis for subsequent modeling of computational latency, communication latency, and energy consumption. Therefore, it is necessary to consider the computational load and communication overhead between subgraphs. This partitioning enables the provision of a dedicated performance evaluation model for each subgraph, thereby accurately calculating various latency and energy consumption in the overall inference process.

[0052] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network is provided, wherein the multimodal neural network includes... There are several levels, and the dividing point is the first dividing point. With the second dividing point The connected subgraph is divided according to the splitting points, resulting in multiple split parts, including: According to the first dividing point With the second dividing point For multiple connected subgraphs Divide into parts to obtain the first part. Part Two and Part Three ; The first part is deployed on terminal devices, the second part is deployed on edge servers, and the third part is deployed on cloud platforms.

[0053] This invention models computational latency, communication latency, and AR terminal power consumption separately. Specifically, it assumes that the multimodal neural network consists of several computational layers, denoted as... ,in This represents the total number of layers in the network. For ease of deployment and computational optimization, use... , This indicates that the entire model is divided into three parts, the first part being... The first part consists of two components, deployed on AR terminals (i.e., terminal devices); the second part consists of... The first part consists of [components], deployed on edge devices (i.e., edge servers); the third part consists of [components]. It consists of components and is deployed on cloud servers (i.e., cloud platforms).

[0054] Through the embodiments of this invention, the hierarchical structure is precisely divided by the splitting points, and only intermediate feature tensors between subgraphs are transmitted (such as the fused features transmitted from the edge to the cloud instead of the original data), significantly reducing the bandwidth pressure of cross-level communication. The terminal device executes the first part (such as single-modal feature extraction), using local computing power to process low-latency sensitive tasks; the edge server hosts the second part (such as cross-modal feature interaction), completing medium-complexity fusion calculations locally; the cloud platform runs the third part (such as high-level semantic decision-making), centrally processing computationally intensive tasks, forming a three-level collaborative architecture of terminal-edge-cloud.

[0055] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network is provided, the method further includes: Based on the first and second split points, the overall inference delay for the first, second, and third parts is determined, including: in, Indicates the first dividing point. Indicates the second dividing point. Indicates overall reasoning delay, Indicates the overall computational delay. This indicates the overall communication delay.

[0056] In this embodiment of the invention, end-to-end inference latency mainly includes two types of overhead: first, the computation latency of each model segment on different nodes; and second, the communication latency generated by intermediate data transmission between heterogeneous nodes. In summary, for a given segmentation scheme... Overall reasoning delay It can be approximated as: in Represents the overall (whole) calculation delay. Represents the overall (total) communication delay.

[0057] Through the embodiments of the present invention, the computational latency and cross-device communication latency of each subgraph at the terminal, edge, and cloud are dynamically evaluated based on the splitting point, avoiding uneven load or transmission bottlenecks caused by empirical splitting.

[0058] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network is provided, the method further includes: Based on the computing power of terminal devices, edge servers, and cloud platforms, the overall computing latency is determined, including: in, Indicates the first dividing point. Indicates the second dividing point. Indicates the overall computational delay. Indicates the first Hierarchical computational load, This indicates the computing power of the terminal device. This indicates the computing power of the edge server. This indicates the computing power of the cloud platform.

[0059] In this embodiment of the invention, for the computational latency, the computational load of the model at the AR terminal (terminal device), edge (edge ​​server), and cloud (cloud platform) execution layers needs to be considered. These computational overheads are closely related to the computational complexity (FLOPs) of each layer and the computing power of the deployed devices, and accumulate in segments throughout the inference process. Floating-point operations are used to represent computational complexity, assuming... Indicates the first The computational load of the layer (in FLOPs), and the computing capabilities of different devices are as follows: , , The total computational latency of the model on the three devices is: Through the embodiments of the present invention, the overall computing latency is decomposed into a weighted sum of three parts: terminal, edge, and cloud. By using the ratio of computing load at each level to device computing power, the actual processing capability of different hardware for subgraphs can be objectively reflected.

[0060] According to the present invention, a cloud-edge-device collaborative deployment method for an image inference multimodal neural network is provided, the method further includes: Based on the output tensor size and communication bandwidth between terminal devices, edge servers, and cloud platforms, the overall communication latency is determined, including: in, Indicates overall communication latency. This indicates that the multimodal neural network is at the first split point. The size of the output feature tensor after segmentation. This indicates that the multimodal neural network is at the second split point. The size of the output feature tensor after segmentation. This represents the communication bandwidth from the terminal device to the edge server. This indicates the communication bandwidth between the edge server and the cloud platform.

[0061] For the communication latency component, the data transmission cost resulting from model splitting across devices needs to be considered. After model splitting, the intermediate feature tensors need to be transferred from one device to another, and the resulting communication overhead is determined by both the tensor size and the communication bandwidth. Let... The model represents the The size of the output feature tensor after segmentation (in bytes), and the communication bandwidth from the AR terminal to the edge server are... The communication bandwidth from the edge server to the cloud server is The data transmission time introduced by model cross-device splitting is defined as: Based on the above modeling metrics, this invention further defines an optimization objective function for collaborative deployment, using task-unit energy efficiency as the metric, in the following form: in For system time constants, Represents overall reasoning delay, This represents the energy consumption of the AR terminal (terminal device). Objective function The larger the value, the higher the efficiency of completing the task under the unit energy consumption of the AR terminal (terminal device), that is, the higher the energy efficiency of completing the task.

[0062] Through the embodiments of the present invention, by establishing the mathematical relationship between the overall inference latency and the size of the feature tensor and the communication bandwidth, the latency contribution of the two key communication paths, terminal-edge and edge-cloud, is clearly revealed, providing data support for network segmentation.

[0063] Step 105: Based on the genetic algorithm, the multimodal neural network is divided into multiple segments, and the divided multimodal neural network is deployed on terminal devices, edge servers, and cloud platforms respectively.

[0064] In some embodiments, the segmentation corresponding to the single-modal processing subgraph is deployed on the terminal device, the segmentation corresponding to the cross-modal interaction subgraph is deployed on the edge server, and the segmentation corresponding to the task-specific subgraph is deployed on the cloud platform.

[0065] refer to Figure 3 , Figure 3 This is a schematic diagram of the multimodal model deployment architecture based on cloud-edge-device collaboration provided by the present invention.

[0066] This invention analyzes the structural characteristics of multimodal neural networks and the inference requirements of power operation and maintenance tasks, and constructs a model decomposition strategy adapted to heterogeneous computing environments. Specifically, the multimodal neural network is divided into multiple continuous sub-segments, and based on the computational load and communication cost of each layer, different sub-segments are rationally deployed on AR terminals, edge nodes, and cloud servers for collaborative execution. For example... Figure 3 As shown, when the input data (text, voice, and images) arrives at the system, the model completes shallow inference at the terminal, and the extracted intermediate features are transmitted to edge nodes with stronger computing power to complete fusion and enhancement. Finally, the cloud completes high-level semantic decision-making, thereby effectively balancing real-time performance and energy consumption.

[0067] The cloud-edge-device collaborative inference includes: an AR terminal, an edge server, and a cloud server. Text input is encoded using an RNN, undergoes feature transformation, and is then input to the (first) multimodal fusion layer. Speech input is processed by a CNN, undergoes feature projection, and is then input to the (first) multimodal fusion layer. Image input undergoes feature extraction using a CNN, undergoes feature enhancement, and is then input to the (first) multimodal fusion layer. Furthermore, the text feature vector obtained from RNN encoding of the text input, the speech feature vector obtained from CNN processing of the speech input, and the visual feature vector obtained from CNN feature extraction of the image input are all input to the (second) multimodal fusion layer to obtain a fused feature representation. The output of the (first) multimodal fusion layer and the fused feature representation are then input to a fully connected decision layer to obtain the output result.

[0068] This invention further proposes an improved genetic algorithm model partitioning optimization method for multimodal deployment, used to automatically search for the optimal partitioning and deployment strategy of multimodal neural networks in a cloud-edge-device heterogeneous computing architecture. This method aims to minimize the latency-energy consumption product, combining subgraph structure partitioning with deployment stage mapping to construct a structured chromosome representation and performance evaluation model, thereby achieving efficient partitioning search under feasibility constraints.

[0069] Specifically, assuming the model is divided into Each subgraph needs to be... Determine a partitioning scheme to form a complete deployment plan. Within each subgraph, a neural network layer sequence is constructed in topological order, and a candidate set is formed by extracting all positions that can serve as valid split points. Each split point is represented by an index, indicating that the subgraph is split into two consecutive layer segments at that location. If each subgraph has an average of If there are 10 candidate split points, then the size of the overall search space is approximately 1000. The solution space grows exponentially. To efficiently explore this solution space, this invention employs a genetic algorithm as the core optimization method. By simulating selection, crossover, and mutation operations in biological evolution, this algorithm can gradually approach the global optimum over multiple generations, greatly improving search efficiency.

[0070] This invention proposes a structured chromosome representation mechanism for multimodal model segmentation. Specifically, a chromosome represents a candidate deployment scheme, consisting of a length of... The vector is composed of three elements, each a triple (block_id, cut_index, stage), representing: the subgraph identifier, the position (index) where the cut occurs, and the deployment position of the starting segment of the subgraph, respectively. During the initial population generation phase, the algorithm selects a cutting layer for each subgraph (or does not cut at all) to form a preliminary chromosome configuration, representing a complete model segmentation deployment scheme. To objectively evaluate the performance of each chromosome (deployment scheme) in a multi-device heterogeneous environment, this invention uses an optimization objective function as the fitness function of the genetic algorithm, in the form: in, For system time constants, Represents overall reasoning delay, It represents the overall energy consumption.

[0071] By maximizing this function, the genetic algorithm can prioritize deployment schemes that significantly reduce terminal overhead while ensuring computational performance.

[0072] During the evolutionary stage, this invention designs genetic operators specifically for deployment optimization, including crossover and mutation operators. The crossover operator randomly selects parent individuals from the population and generates new individuals by pairing and exchanging chromosome segments according to their subgraph positions, ensuring the validity of the solution. The mutation operator perturbs a gene in the chromosome with a certain probability, including changing the split point or adjusting the deployment phase. This mutation operation is constrained by structural validity, ensuring that the model topology and deployment continuity are still satisfied after the operation. Finally, individuals are ranked based on a fitness function, and the top performers are selected. The optimal solution is used as the next generation parent, until the maximum number of iterations is reached or the convergence condition is met.

[0073] To ensure that all generated individuals are legal and valid, the algorithm needs to use a legality verification module to check chromosomes in each generation of evolution, ensuring that candidate deployment strategies meet the requirements of legal splitting order, coherent deployment phases, and reachable cross-phase data dependencies.

[0074] Throughout the iterative process, the algorithm evolves continuously using the classic iterative framework of "initialization-evaluation-selection-crossover-mutation-screening". In each generation, the algorithm selects individuals according to the fitness roulette strategy, that is, it probabilistically guides the transmission of more high-fitness individuals to the next generation; through crossover and mutation, a new generation of chromosomes is generated, and selection is carried out in combination with legitimacy verification, continuously updating the population to obtain a better deployment strategy.

[0075] refer to Figure 4 , Figure 4 This is a flowchart of the genetic algorithm for model segmentation deployment provided by the present invention.

[0076] Figure 4 This paper demonstrates the evolutionary optimization algorithm flow for multimodal neural network subgraph segmentation deployment. Inputs include the model subgraph topology, candidate segmentation point set, device performance parameters, and bandwidth. During initialization, segmentation points are randomly selected and chromosomes are derived (encoding format: (block_id, cut_index, stage)). Chromosome validity must be verified (ensuring the DAG topology aligns with the terminal-edge-cloud execution order). Fitness evaluation is performed using a formula. The algorithm comprehensively considers latency and energy consumption. The main loop uses selection operations (using roulette wheel to select individuals with high fitness), structured crossover (exchanging split points and constructing new chromosomes), and structured mutation (randomly modifying split points and constructing new chromosomes). After each operation, the legality of the chromosomes must be verified (checking dependencies, ensuring the execution order of terminal-edge-cloud, filtering illegal individuals, and dependency constraints: strictly following the DAG topology) to form a new population. The termination condition is reaching the maximum number of iterations or fitness convergence. If these conditions are met, the optimal split deployment scheme is output.

[0077] This method, through a population evolution mechanism, effectively preserves historically high-performing strategy fragments, gradually approaching the global optimum during the search process and avoiding getting trapped in local optima. Furthermore, genetic algorithms inherently support multi-objective optimization, enabling them to balance various constraints while simultaneously considering latency and energy consumption requirements, resulting in solutions with greater practical deployment value. Simultaneously, the algorithm can flexibly adapt to actual hardware performance and network conditions during deployment, further enhancing the system's robustness and adaptability.

[0078] In summary, the model deployment optimization method based on genetic algorithms proposed in this invention can automatically search for collaborative deployment strategies with optimal latency-energy consumption performance on the basis of the complex structure of multimodal neural networks, providing important support for building efficient and low-energy distributed intelligent inference systems.

[0079] The key to this invention lies in proposing an efficient segmentation and deployment method for multimodal neural networks. Through collaborative optimization across a heterogeneous computing architecture spanning cloud, edge, and terminal layers, it effectively addresses the real-time performance and energy efficiency challenges in power system operation and maintenance (PSG) scenarios using AR. Specifically, this invention employs a structured subgraph partitioning strategy to divide complex multimodal models into several functional subgraphs and intelligently allocates tasks among cloud, edge, and terminal devices, thereby maximizing the utilization efficiency of computing resources and reducing communication latency and energy consumption. This method combines global optimization with a genetic algorithm, enabling rapid approximation of the optimal solution in a complex nonlinear combinatorial space. It achieves multi-objective optimization (including latency and terminal energy consumption), ensuring adaptability to performance differences across different hardware platforms in dynamic deployment environments, thus effectively improving the intelligence level and operational efficiency of AR applications in power system operation and maintenance.

[0080] This invention significantly improves the processing efficiency and intelligence level of power operation and maintenance tasks in AR scenarios by optimizing the segmentation and deployment of multimodal models through cloud-edge-device collaboration. Intelligent task allocation and dynamic adaptation successfully reduce communication latency and energy consumption while ensuring the system's efficiency and stability when facing complex computational tasks. The introduction of genetic algorithms enables the rapid finding of optimal solutions in large-scale nonlinear combinatorial spaces, effectively avoiding the dilemma of local optima and providing faster and more energy-efficient technical support for real-time decision-making and on-site operations in power operation and maintenance. Overall, this invention not only improves the real-time response capability of AR applications in power operation and maintenance but also enhances the system's flexibility and scalability, providing a practical solution for the intelligent development of the industry.

[0081] The following describes an example of the cloud-edge-device collaborative deployment method for image inference multimodal neural networks provided by this invention in a practical application.

[0082] To verify the feasibility and performance advantages of a multimodal model deployment optimization method based on genetic algorithms, this invention constructs a three-layer cloud-edge-device collaborative inference system in a simulated environment and evaluates the performance of a custom multimodal neural network. In the experiment, a visual-audio multimodal fusion model was used. This model performs feature fusion through a bidirectional cross-attention module and ultimately outputs the prediction result. The input data includes an image of size 1×3×224×224 and an audio clip of size 1×64×160, simulating real-world device data.

[0083] The system architecture comprises terminal devices, edge computing nodes, and cloud servers, simulating the computing and communication capabilities of smartphones, AR glasses, and cloud servers, respectively. The terminal devices have 100 GFLOPs of computing power, the edge devices 3 TFLOPs, and the cloud devices 100 TFLOPs. For communication, the bandwidth between the terminal and the edge is set to 300 Mbps, and the bandwidth between the edge and the cloud is set to 1000 Mbps. Experiments quantify the latency and energy consumption of each device during task execution to evaluate the performance of different partitioning strategies. To verify the advantages of the method, comparisons were made with full-terminal deployment, fixed partitioning points, and random partitioning deployment strategies, focusing on analyzing inference latency, energy consumption, and the rationality of the deployment structure. Through a deployment scheme optimized by a genetic algorithm, the system achieved a significant performance improvement.

[0084] In the experimental comparison, we first analyzed the performance of different deployment strategies in terms of total inference latency.

[0085] refer to Figure 5 , Figure 5 This is a comparison of latency for different deployment strategies provided by this invention.

[0086] like Figure 5 As shown, the latency of deploying the complete model on the terminal device (terminal_only) is similar to that of random deployment without a strategy (random), which is about 40 milliseconds; while the optimal splitting strategy (ga_best) obtained in this invention effectively compresses the inference latency to 21.6 milliseconds, reducing the latency by about 46.8%.

[0087] refer to Figure 6 , Figure 6 This invention provides a comparison of the energy consumption of terminal devices under different deployment strategies.

[0088] From the perspective of terminal energy consumption, such as Figure 6 As shown, having a large number of high-computation-density modules running on the terminal will result in high energy consumption; the energy consumption of the random and terminal_only strategies is 2.41 × 10⁻⁶ respectively. 8 nJ and 2.45×10 8 The segmentation structure obtained in this invention is more biased towards migrating high-FLOPs modules to the edge or cloud side in terms of node distribution, which greatly reduces the burden on the terminal battery and reduces energy consumption to approximately 5.04 × 10⁻⁶. 7 The nJ value is reduced by approximately 79%-80% compared to the baseline. This result validates the high adaptability of this method in deployment tasks for "energy-constrained devices".

[0089] refer to Figure 7 , Figure 7 This is a comparison of the adaptability of different deployment strategies provided by this invention.

[0090] The optimization effect can be further verified by calculating the objective function results for different deployment structures and calculating the fitness index. For example... Figure 7 As shown, the fitness value obtained by ga_best is approximately 9.19 × 10⁻ 7 This is significantly higher than random (1.02 × 10⁻⁻⁶). 7 ) and terminal_only (1.00×10⁻ 7 This indicates that the optimization strategy has significant advantages under the global resource (computing power, bandwidth utilization, energy consumption cost) scheduling objective.

[0091] In further analyzing the evolution of deployment strategies, this invention focuses on the impact of actual terminal device computing power on the deployment structure segmentation depth. Tests showed that the model segmentation points selected by the genetic algorithm during the search process changed significantly under different terminal computing power constraints.

[0092] refer to Figure 8 , Figure 8 This refers to the relative depth of the segmentation position under different terminal device computing capabilities provided by the present invention.

[0093] like Figure 8 As shown, with the increase in terminal computing power, the relative depth of the two key splitting points, Split 1 (Terminal→Edge) and Split 2 (Edge→Cloud), gradually increases in the overall computational path of the model. When the terminal computing power is weak, Split 1 is relatively close to the model entry point, only undertaking the initial preprocessing module tasks, while the main computational tasks are completed on subsequent servers. As the terminal computing power gradually increases, Split 1 migrates significantly backward, meaning that more model modules are redeployed back to the terminal side, indicating that the genetic algorithm automatically identifies terminal resource redundancy and utilizes it efficiently. At the same time, the splitting point of Split 2 is also adjusted synchronously. The trend in this figure shows that the deployment splitting strategy driven by the genetic algorithm is highly correlated with resource dynamics, possessing good structural sensitivity and topology adaptability.

[0094] In summary, the experimental metrics clearly demonstrate that the proposed evolutionary search-based model collaborative deployment strategy exhibits significant advantages over traditional methods in terms of latency and energy consumption optimization. These advantages are not only reflected in the final performance optimization but also in its good scalability and adjustability during structural optimization and algorithm scheduling. Therefore, this method is applicable to generalized scenarios in multimodal neural network collaborative deployment problems, and is particularly suitable for real-world environments with complex structures and heterogeneous resource constraints.

[0095] The cloud-edge-device collaborative deployment system for image inference multimodal neural networks provided by this invention will be described below. The cloud-edge-device collaborative deployment system for image inference multimodal neural networks described below can be referred to in correspondence with the cloud-edge-device collaborative deployment method for image inference multimodal neural networks described above.

[0096] refer to Figure 9 , Figure 9 This is a schematic diagram of the cloud-edge-device collaborative deployment system for image inference multimodal neural networks provided by the present invention.

[0097] Abstraction module 901 is used to abstract a preset multimodal neural network to obtain an original computation graph, wherein the original computation graph is a directed acyclic graph and includes a set of computation nodes and a set of data dependency edges. The splitting module 902 is used to split the original computation graph based on the set of computation nodes and the set of data dependency edges to obtain multiple connected subgraphs, wherein the categories of the multiple connected subgraphs include: single-modal processing subgraphs, cross-modal interaction subgraphs, and task-specific subgraphs; The determining module 903 is used to determine the splitting point of each connected subgraph based on the topological order of the multiple connected subgraphs; The partitioning module 904 is used to partition the plurality of connected subgraphs according to the partitioning points to obtain a plurality of partitioned parts; The deployment module 905 is used to divide the multimodal neural network as a whole based on the multiple segmentation parts according to the genetic algorithm, and to deploy the multimodal neural network after the overall division on terminal devices, edge servers and cloud platforms respectively.

[0098] Specifically, the cloud-edge-device collaborative deployment system for image inference multimodal neural networks provided by the present invention can implement all the method steps implemented in the above-mentioned cloud-edge-device collaborative deployment method embodiment for image inference multimodal neural networks, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0099] Figure 10 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 10As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communications bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other through the communications bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a cloud-edge-device collaborative deployment method for image inference multimodal neural networks. This method includes: abstracting a preset multimodal neural network to obtain an original computation graph, wherein the original computation graph is a directed acyclic graph and includes a set of computation nodes and a set of data-dependent edges; splitting the original computation graph based on the set of computation nodes and the set of data-dependent edges to obtain multiple connected subgraphs, wherein the categories of the multiple connected subgraphs include: single-modal processing subgraphs, cross-modal interaction subgraphs, and task-specific subgraphs; determining the splitting point of each connected subgraph based on its topological order; dividing the multiple connected subgraphs according to the splitting point to obtain multiple segmented parts; and using a genetic algorithm to perform an overall partitioning of the multimodal neural network based on the multiple segmented parts, and deploying the overall partitioned multimodal neural network on terminal devices, edge servers, and cloud platforms respectively.

[0100] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cloud-edge-device collaborative deployment method for image inference multimodal neural networks provided by the above methods. The method includes: abstracting a preset multimodal neural network to obtain an original computation graph, wherein the original computation graph is a directed acyclic graph and includes a set of computation nodes and a set of data-dependent edges; splitting the original computation graph based on the set of computation nodes and the set of data-dependent edges to obtain multiple connected subgraphs, wherein the categories of the multiple connected subgraphs include: single-modal processing subgraphs, cross-modal interaction subgraphs, and task-specific subgraphs; determining the splitting point of each connected subgraph based on the topological order of the multiple connected subgraphs; dividing the multiple connected subgraphs according to the splitting point to obtain multiple split parts; performing an overall partitioning of the multimodal neural network based on the multiple split parts using a genetic algorithm, and deploying the overall partitioned multimodal neural network on terminal devices, edge servers, and cloud platforms respectively.

[0102] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a cloud-edge-device collaborative deployment method for image inference multimodal neural networks provided by the above methods. The method includes: abstracting a preset multimodal neural network to obtain an original computation graph, wherein the original computation graph is a directed acyclic graph and includes a set of computation nodes and a set of data-dependent edges; splitting the original computation graph based on the set of computation nodes and the set of data-dependent edges to obtain multiple connected subgraphs, wherein the categories of the multiple connected subgraphs include: single-modal processing subgraphs, cross-modal interaction subgraphs, and task-specific subgraphs; determining the splitting point of each connected subgraph based on the topological order of the multiple connected subgraphs; dividing the multiple connected subgraphs according to the splitting point to obtain multiple split parts; performing an overall partitioning of the multimodal neural network based on the multiple split parts using a genetic algorithm, and deploying the overall partitioned multimodal neural network on terminal devices, edge servers, and cloud platforms respectively.

[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud-edge-device collaborative deployment method for image inference multimodal neural networks, characterized in that, include: The pre-defined multimodal neural network is abstracted to obtain the original computation graph, wherein the original computation graph is a directed acyclic graph and includes: a set of computation nodes and a set of data dependency edges; Based on the set of computing nodes and the set of data-dependent edges, the original computing graph is split to obtain multiple connected subgraphs, wherein the categories of the multiple connected subgraphs include: single-modal processing subgraphs, cross-modal interaction subgraphs, and task-specific subgraphs; Based on the topological order of the multiple connected subgraphs, determine the splitting point of each connected subgraph; The multiple connected subgraphs are divided according to the stated dividing points to obtain multiple segmented parts; The multimodal neural network is divided into multiple segments based on the genetic algorithm, and the resulting multimodal neural network is deployed on terminal devices, edge servers, and cloud platforms.

2. The cloud-edge-device collaborative deployment method for image inference multimodal neural networks according to claim 1, characterized in that, The original computation graph is split based on the set of computation nodes and the set of data-dependent edges to obtain multiple connected subgraphs, including: Based on the set of computing nodes and the set of data dependency edges, the data connection relationship and semantic function of each node are determined; Based on the data connection relationship and the semantic function, the substructure located at the front end of the multimodal neural network that only receives a single modal input stream is identified and divided to obtain a single modal processing subgraph; Based on the data connection relationship, the data fusion substructure in the multimodal neural network that simultaneously receives input from at least two modalities is located and divided to obtain a cross-modal interaction subgraph; Based on the semantic functions, the substructures located downstream of the multimodal neural network and used for decision prediction or generation tasks are identified and divided to obtain task-specific subgraphs.

3. The cloud-edge-device collaborative deployment method for image inference multimodal neural networks according to claim 1, characterized in that, The multimodal neural network includes There are several levels, and the dividing points are respectively the first dividing point. With the second dividing point The step of dividing the multiple connected subgraphs according to the dividing points to obtain multiple dividing parts includes: According to the first division point With the second dividing point For the multiple connected subgraphs Divide into parts to obtain the first part. Part Two and Part Three ; The first part is deployed on the terminal device, the second part is deployed on the edge server, and the third part is deployed on the cloud platform.

4. The cloud-edge-device collaborative deployment method for image inference multimodal neural networks according to claim 3, characterized in that, The method further includes: Based on the first and second split points, the overall inference delay of the first part, the second part, and the third part is determined, including: in, This indicates the first dividing point. Indicates the second dividing point. This indicates the overall reasoning delay. Indicates the overall computational delay. This indicates the overall communication delay.

5. The cloud-edge-device collaborative deployment method for image inference multimodal neural networks according to claim 4, characterized in that, The method further includes: Based on the computing power of the terminal device, the computing power of the edge server, and the computing power of the cloud platform, the overall computing latency is determined, including: in, This indicates the first dividing point. Indicates the second dividing point. This indicates the overall computational delay. Indicates the first Hierarchical computational load, This indicates the computing power of the terminal device. This indicates the computing power of the edge server. This indicates the computing power of the cloud platform.

6. The cloud-edge-device collaborative deployment method for image inference multimodal neural networks according to claim 4, characterized in that, The method further includes: The overall communication latency is determined based on the output tensor size and communication bandwidth between the terminal device, the edge server, and the cloud platform, including: in, This indicates the overall communication delay. This indicates that the multimodal neural network is at the first split point. The size of the output feature tensor after segmentation. This indicates that the multimodal neural network is at the second split point. The size of the output feature tensor after segmentation. This represents the communication bandwidth from the terminal device to the edge server. This indicates the communication bandwidth from the edge server to the cloud platform.

7. A cloud-edge-device collaborative deployment system for image inference multimodal neural networks, characterized in that, include: An abstraction module is used to abstract a preset multimodal neural network to obtain an original computation graph, wherein the original computation graph is a directed acyclic graph and includes a set of computation nodes and a set of data dependency edges. The splitting module is used to split the original computation graph based on the set of computation nodes and the set of data dependency edges to obtain multiple connected subgraphs, wherein the categories of the multiple connected subgraphs include: single-modal processing subgraphs, cross-modal interaction subgraphs, and task-specific subgraphs; The determination module is used to determine the splitting point of each connected subgraph based on the topological order of the multiple connected subgraphs; The partitioning module is used to partition the multiple connected subgraphs according to the partitioning points to obtain multiple partitioned parts; The deployment module is used to divide the multimodal neural network into its entirety based on the multiple segmentation parts using a genetic algorithm, and then deploy the divided multimodal neural network onto terminal devices, edge servers, and cloud platforms.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the cloud-edge-device collaborative deployment method for the image inference multimodal neural network as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cloud-edge-device collaborative deployment method of the image inference multimodal neural network as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the cloud-edge-device collaborative deployment method of the image inference multimodal neural network as described in any one of claims 1 to 6.

Images