Edge model scheduling method and system fusing graph nerve and hybrid expert model

Abstract

The invention belongs to the field of computer artificial intelligence and wireless communication networks, and provides an edge model scheduling method and system fusing graph nerves and a hybrid expert model, and the method comprises the steps: collecting multi-modality, hardware parameters and model performance data, constructing a graph nerve network structure according to the resource condition of edge equipment and task calculation demands, and carrying out the scheduling of the edge model. Therefore, the topological relation and conditional dependence among data, hardware and models are captured; according to the method, feature processing is carried out at the local where data is generated, so that the time delay of data transmission to a cloud or a central server is remarkably reduced, and the resource utilization and model processing efficiency of the whole system is improved; the graph neural network prediction model constructed based on the general features and the new message passing mechanism can effectively deal with the zero sample prediction problem in an unfamiliar hardware equipment environment, does not need to perform special training and optimization for each hardware equipment and generate diversified prediction results, and effectively reduces the risk of prediction result overfitting.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer artificial intelligence and wireless communication network, and in particular relates to an edge model scheduling method and system that integrates graph neural network and hybrid expert model. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Large models have made remarkable progress in the practice of multiple application fields such as computer vision, natural language processing, and big data analysis. With the rapid development of emerging technologies such as the Internet of Things (IoT) and 5G communications, the deployment and reasoning needs of large models in cloud-edge environments are becoming increasingly urgent. However, the application of large model technology in edge scenarios such as the Industrial Internet is limited by complex and diverse application scenarios and equipment conditions. There are still many challenges in implementing the deployment and operation of large models in edge environments. In order to meet the computing resource requirements of large models, a common practice is to transmit data with large-scale servers far away from the data source through network communication, so as to assist in processing data and training models with the help of cloud computing resources. However, this method has problems such as transmission delay and privacy security. Among them, one of the main challenges is to meet the high resource requirements of deep learning on weaker edge computing resources. These devices often have relatively weak CPU and GPU performance and limited storage capacity, which cannot be compared with high-performance servers in the cloud. Another is how to deploy a trained large model to an actual zero-shot application environment, which means that the model needs to recognize and classify parameters and scenarios that have never been seen before. The hardware platforms in different actual application environments have different parameters and structures, and the models may face compatibility issues, resulting in failure to operate normally or performance degradation. How to efficiently manage and allocate edge resources and ensure the success rate of large models after the first deployment, especially how to predict and optimize the placement of large models in complex edge environments, ensure the efficiency and quality of completing tasks to ensure good end-to-end applications, has also become a difficult challenge in the field of edge computing. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes an edge model scheduling method and system that integrates graph neural networks and hybrid expert models. The present invention can effectively solve the placement problem of large language models in edge environments in the prior art, so that the expert models in the multimodal hybrid expert model can be reasonably and efficiently distributed to different edge devices, thereby ensuring the successful deployment and stable operation of large language models in heterogeneous edge devices, and improving the running speed and load balance of artificial intelligence models in resource-constrained environments.

[0005] According to some embodiments, a first solution of the present invention provides an edge model scheduling method integrating graph neural network and hybrid expert model, which adopts the following technical solution: The edge model scheduling method that integrates graph neural network and hybrid expert model includes: Preprocess the multimodal data, and generate a multimodal hybrid expert model based on the preprocessed multimodal data; Obtain edge device hardware parameters and multimodal data feature splicing to generate a general feature data set; Based on the common feature dataset, a heuristic algorithm is used to enumerate and generate multiple candidate placement schemes of the hybrid expert model, and multiple feasible candidate placement schemes are screened out according to the hardware constraints of the edge device; Based on the common feature dataset, the expert model and the corresponding edge devices are set as independent nodes with feature attributes by using feature embedding method and message passing mechanism, and edge connections are established according to feasible candidate placement schemes to construct the edge device relationship graph neural network. A cost function is established according to the resource status of the edge device and the computing requirements of the model. The edge device relationship graph neural network is iteratively trained to find candidate placement solutions with the goal of minimizing the cost function value, and the optimal placement solution of the hybrid expert model is obtained.

[0006] Furthermore, the processing process of the hybrid expert model is specifically as follows: Obtain multimodal data for standardization and feature extraction, and use feature embedding to map multimodal data into a unified feature space; Generate modal tokens corresponding to multiple modalities from the unified feature space to obtain multiple modal tokens; Using gated routing, the expert models are sorted in descending order of their weights, and multiple modal tokens are assigned to a corresponding number of expert models with high weight values ​​and not fully occupied for processing; The output results of multiple expert models are fused with the multiplication results of their weights and the original modal tokens to obtain a fusion result, which is then normalized and decoded as the final output result of the hybrid expert model.

[0007] Furthermore, based on the general feature data set, a heuristic algorithm is used to enumerate and generate multiple candidate placement schemes of the hybrid expert model, and multiple feasible candidate placement schemes are screened out according to the hardware constraints of the edge device, specifically: Initialize the edge device relationship graph neural network and assign values ​​according to the type and characteristics of each node to generate multiple model nodes and hardware nodes, where the model nodes include data nodes, routing nodes, expert nodes, and output nodes; Setting constraints on hardware nodes based on the eigenvalues ​​in each node; After sorting the expert nodes in random and linear sequence, the model nodes and hardware nodes are combined and allocated using a heuristic enumeration method to generate multiple candidate placement schemes for the hybrid expert model; The validity of candidate placement solutions is judged based on hardware constraints, infeasible candidate placement solutions are filtered out, and multiple feasible candidate placement solutions are screened out.

[0008] Furthermore, the placement process of the hardware nodes is combined and allocated with the model nodes according to the following three rules, specifically: Each hardware node receives at least one model node; Hardware nodes are arranged from weak to strong based on resources and performance; During the placement process, model nodes are placed into hardware nodes in the order of data flow in a line.

[0009] Furthermore, the construction process of the edge device relationship graph neural network is specifically as follows: The feature embedding method is used to map the general feature data set into a continuous vector space to obtain a feature vector set; According to the candidate placement scheme, the corresponding feature vector is sequentially transmitted from the expert node, data node, routing node and output node to the hardware node assigned to it; Then reversely transmit the characteristic information from the hardware node to the associated node; The feature information is passed in the order of the data flow in which the modal tokens are processed in each expert model run; In this way, edges are constructed according to the relationship between each node to obtain the edge device relationship graph neural network.

[0010] Furthermore, the cost function includes the amount of data processed by the expert model per unit time, the load balance of the edge nodes, and the communication time of the expert model for processing the modal token.

[0011] According to some embodiments, a second solution of the present invention provides an edge model scheduling system integrating graph neural network and hybrid expert model, which adopts the following technical solution: The edge model scheduling system that integrates graph neural networks and hybrid expert models includes: A hybrid expert model generation module is configured to preprocess the multimodal data and generate a multimodal hybrid expert model based on the preprocessed multimodal data; A feature stitching module is configured to obtain edge device hardware parameters and multimodal data feature stitching to generate a general feature data set; A feasible candidate placement scheme determination module is configured to enumerate and generate multiple candidate placement schemes of a hybrid expert model based on a common feature data set using a heuristic algorithm, and screen out multiple feasible candidate placement schemes according to hardware constraints of edge devices; The graph neural network building module is configured to use a feature embedding method and a message passing mechanism to set the expert model and the corresponding edge device as independent nodes with feature attributes based on a common feature data set, and to establish edge connections according to feasible candidate placement solutions to build a graph neural network for edge device relationships; The optimal placement plan determination module is configured to establish a cost function based on the resource status of the edge device and the computing requirements of the model, iteratively train the edge device relationship graph neural network to obtain the candidate placement plan with the goal of minimizing the cost function value, and obtain the optimal placement plan of the hybrid expert model.

[0012] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0013] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the edge model scheduling method of fusing graph neural and hybrid expert models as described in the first aspect above.

[0014] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0015] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the edge model scheduling method of fusing graph neural and hybrid expert models as described in the first aspect above are implemented.

[0016] According to some embodiments, a fifth aspect of the present invention provides a computer program product or a computer program.

[0017] The present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the edge model scheduling method of the fusion graph neural and hybrid expert model as described in the first aspect above.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention is aimed at application scenarios of timely and efficient processing of multimodal data in the industrial Internet environment. It can make full use of edge node resources with limited local computing power, directly perform feature processing at the local location where the data is generated, and reduce the time delay of transmitting data to the cloud or central server, thereby improving the calculation speed and operation efficiency of the multimodal hybrid expert model. At the same time, it can also make full use of the computing resources of the edge nodes to improve the resource utilization and model processing efficiency of the overall system. The present invention can deal with zero-shot prediction problems in hardware device environments that have never been seen before, that is, beyond the scope of the training data set. Under zero-shot conditions, there is no need to conduct special training and optimization for each hardware device, thereby improving the applicability and flexibility of the expert model, improving the success rate and response speed of the operation based on the multimodal hybrid expert model when it is placed in an unfamiliar edge environment for the first time, and reducing the complexity and cost of the preliminary preparation work for model deployment; The GNN zero-shot prediction model based on the new message passing mechanism in the present invention uses a heuristic enumeration algorithm to iteratively obtain allocation plans defined based on multiple cost parameters, generate diversified prediction results for the same model execution cycle, and effectively reduce the risk of overfitting of the prediction results. Each node of the model can better integrate the complex relationships and feature information in the overall graph neural network, which helps to effectively search and identify the most cost-effective allocation strategy in a complex decision space. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0020] Figure 1 It is a flow chart of an edge model scheduling method integrating graph neural and hybrid expert models in an embodiment of the present invention; Figure 2 It is a heuristic enumeration process in an embodiment of the present invention; Figure 3 is a flow chart of a feature embedding method (Feature Embedding) in an embodiment of the present invention; Figure 4 is a flow chart of a new message passing mechanism (Massage Passing) in an embodiment of the present invention; Figure 5 is a structural diagram of a multimodal mixture of experts model (MMoE) in an embodiment of the present invention; Figure 6 It is a deployment and implementation flow chart in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0023] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0024] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0025] Embodiment 1 The present embodiment provides an edge model scheduling method that integrates graph neural and hybrid expert models. The present embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal, a server, and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In the present embodiment, the method includes the following steps: Preprocess the multimodal data, and generate a multimodal hybrid expert model based on the preprocessed multimodal data; Obtain edge device hardware parameters and multimodal data feature splicing to generate a general feature data set; Based on the common feature dataset, a heuristic algorithm is used to enumerate and generate multiple candidate placement schemes of the hybrid expert model, and multiple feasible candidate placement schemes are screened out according to the hardware constraints of the edge device; Based on the common feature dataset, the expert model and the corresponding edge devices are set as independent nodes with feature attributes by using feature embedding method and message passing mechanism, and edge connections are established according to feasible candidate placement schemes to construct the edge device relationship graph neural network. A cost function is established according to the resource status of the edge device and the computing requirements of the model. The edge device relationship graph neural network is iteratively trained to find candidate placement solutions with the goal of minimizing the cost function value, and the optimal placement solution of the hybrid expert model is obtained.

[0026] like Figure 1 As shown, an edge model placement prediction algorithm based on graph neural network and multimodal hybrid expert model includes the following steps: Step (1): Generate a dataset consisting of common features, such as the number of CPUs, transmission bandwidth, etc., based on edge hardware parameters and multimodal data; Step (2): Generate multiple candidate placement solutions based on heuristic enumeration and determine the feasibility of the solutions based on hardware constraints; Step (3): Using the overall graph structure constructed by the feature dataset and candidate placement schemes, with the help of feature embedding method and new message passing mechanism, the expert model and the corresponding edge device are set as independent nodes with feature attributes, and edge connections are established according to the placement scheme; Step (4): Based on the generated graph neural network, establish a training data set and a loss function based on parameters such as operation success rate and IO latency, iteratively train the GNN model, and generate prediction values ​​to select the best placement solution.

[0027] The specific process is as follows: In step (1), the meaning and content of the data set composed of common features include: In order to better use the model to predict the running cost and predict the pros and cons of the placement plan, the common features must best reflect the attributes of the running cost, such as hardware parameters, data structure and other parameters.

[0028] On the one hand, universal features mean wider adaptability in different edge devices and multimodal data. This allows the same model or algorithm to run on multiple devices without the need for special optimization and adjustment for each device, thereby improving the success rate of model deployment. Moreover, universal features can map features from data of different modalities (such as image data, text data, audio data, etc.) into the same feature space, that is, the multimodal feature space, to achieve the fusion and unification of multimodal data, helping the model to better understand and process complex information, so as to migrate and apply it in different tasks.

[0029] On the other hand, when deploying the trained model to the actual application environment, it is necessary to consider how to deal with data content that is different from the training data, that is, the generalization ability of the model. Using common features to train the model can have better generalization performance, because common features can often better reflect the core patterns and rules, and can also simplify the overall complexity to a certain extent, have better robustness to noise interference in real scenes and transferability to cope with various tasks, thus helping the model to adapt to various features across different scenes and data.

[0030] Table 1 below is the common features used in the present invention, which are used to initialize the parameters of each node in the graph neural network: Table 1 Graph node characteristics

[0031] In step (2), the specific steps for generating multiple candidate placement solutions based on heuristic enumeration and judging the feasibility of the solutions based on hardware constraints are as follows. The overall process is as follows: Figure 2 As shown: Initialize the graph neural network and assign values ​​according to the node types and features in Table 1 to generate multiple model nodes and hardware nodes; the model nodes include data nodes, routing nodes, expert nodes, and output nodes Set constraints on hardware nodes based on the characteristic values ​​in each node, such as whether the resource parameters of the hardware nodes can meet the storage and computing requirements of the model node parameters, whether the transmission speed between hardware nodes does not exceed the preset threshold, etc. After sorting the expert nodes in random and linear sequences, the model nodes and hardware nodes are combined and allocated using a heuristic enumeration method. At the same time, the placement process of the hardware nodes is ensured to comply with the following three rules, and multiple candidate placement schemes for the hybrid expert model are obtained: Each hardware node receives at least one model node, that is, multiple model nodes can be assigned to the same hardware node to ensure the independence of each expert model; Hardware nodes are arranged from weak to strong according to resources and performance to ensure load balancing among multiple nodes; During the placement process, the model nodes are placed in the hardware nodes in the order of data flow in a line. Specifically, during the placement process, the model nodes are placed in the hardware nodes in the order of data flow from "data node" to "routing node" to "expert node" to "output node". There is no data loop, thereby reducing the complexity of data flow and the overhead of network communication.

[0032] The effectiveness of candidate placement solutions is preliminarily determined based on hardware constraints, infeasible candidate placement solutions are filtered out, and multiple feasible candidate placement solutions are screened out.

[0033] In step (3), based on the above steps, multiple feasible candidate placement solutions are obtained. The present invention designs and adopts a feature embedding method (Feature Embedding) and a new message passing mechanism (Massage Passing) to help nodes obtain feature information of neighboring nodes and the global network. The process is as follows: Figure 3 and Figure 4 As shown: Feature Embedding is used to map high-dimensional sparse features into a continuous vector space, thereby retaining the core semantic information of the data, better reflecting the implicit relationship and distribution law between features, and improving the generalization ability of the model. The present invention designs a new message passing mechanism (Massage Passing) to iteratively update the special information of each independent node multiple times, and updates the parameters in the central node with the information in the adjacent nodes by parameter aggregation, so that the node contains the characteristic information of the adjacent nodes and the global network, and obtains the relevant characteristic information according to the data flow situation. The specific steps are as follows: According to the feasible candidate placement scheme in step (2), the feature information is transmitted from the expert node, data node, router node, and end node to the hardware node assigned to it; The characteristic information of the hardware node is transmitted to the associated nodes in the reverse direction of the previous step; The feature information is transmitted in the order of the token data stream processed in each MMoE model run; After the above update is completed, the feature information of all nodes is integrated in the final multi-layer perceptron layer (Final MLPLayer), and the resulting results will be used to predict the running cost of the MMoE model using the candidate placement solution to predict the best placement solution.

[0034] In step (4), based on the graph neural network generated in the above steps, a loss function based on parameters such as operation success rate and delay is established, and the GNN model is iteratively trained using the training data set to generate the MMoE model cost prediction value to select the best placement solution. The process is as follows: Figure 2 As shown: In order to accurately reflect the effect and performance of the model placement solution, the present invention defines a model cost function representing the graph neural network: (1); Throughput ( Throughout , ) refers to the amount of data that the MMoE model can process per unit time. For a given query execution, throughput is defined as the number of tokens that arrive at the output node per time unit, expressed in tokens / s (the number of tokens processed per second); Load Balancing ( Balance , ) is used to select a solution that can maximize the opportunities for edge hardware nodes to obtain balanced data processing during model application, thereby avoiding over-calling of certain edge hardware devices or neglect of certain devices due to significant data category imbalance, helping the model to better integrate multimodal data information and improve the accuracy and robustness of video understanding. is the number of experts, It is The probability of an expert being selected is: (2); Processing time ( Processing Time , ) represents the time required from when a token is passed into a hardware node where an expert model is located until the processed data is transmitted from the node; Communication time ( Communication Time , ) represents the sum of all the time required for the token to be transmitted between nodes , including the transmission time ( Transfer Time , ) and queuing time ( Queue Time , ), is the number of experts: (3); Placement successful ( Placement Success , ) indicates the success of the MMoE model operation: (4); The GNN model is trained and iterated according to multiple cost function parameters. The loss function used here is the mean log square error (MSLE), which contains the actual value and predicted values , the formula is as follows: (5); The GNN model generated through multiple iterations generates prediction results for multiple MMoE model candidate placement schemes and conducts comprehensive analysis, and finally selects the optimal MMoE model placement scheme.

[0035] Iterative training of the GNN model is to iteratively process different feasible candidate placement solutions. A trained MMoE model is required first. GNN is divided into two processes: training and running: Training is to use the loss function to generate multiple candidate solutions based on the training data including hardware nodes and MMoE model parameters, and iteratively optimize the model until a good prediction result is formed; The operation is to use the trained GNN model to generate an optimal placement solution result based on the hardware nodes and MMoE model parameters that need to be predicted.

[0036] like Figure 5 As shown in Figure 1, a complete execution process of the multimodal hybrid expert model is as follows: Multimodal data collection, including image data, text data, and audio data; With the help of encoders for standardization and feature extraction, the multimodal data is feature-mapped to a unified multimodal feature space through data embedding and multiple tokens are generated. Each execution process processes a token content. Using gated routing, data is allocated to k experts with high weight values ​​and not fully occupied according to the descending order of weight given by the expert model, that is, these k experts have better performance on the current token than other experts. Among them, the structure of the expert model (MoE) is unified as a feedforward neural network (FFN), and the main difference is that the parameters learned during the training process are different; Using residual connection and normalization methods, the output results of multiple expert models are combined with the multiplication results of their weights and the original token. Assume that the input token is , the gated routing layer calculates The expert weight is , No. The output of the expert model is , the final output result can be expressed as: (6); Normalize and decode the fused multimodal data information into the final output result; According to the above content, labels and parameter values ​​are set to facilitate the construction and prediction of graph neural networks.

[0037] like Figure 6 As shown, this embodiment aims to solve the placement problem of the multimodal hybrid expert model in the edge environment and improve the running speed and load balance of the model in a resource-constrained environment. The following is a specific implementation method of the present invention deployed and run on an edge device: Data collection and preprocessing. First, determine the multimodal task objectives, use sensors and input devices to obtain and store multimodal data from edge devices, including original text data, original image data, and original audio data; obtain hardware parameter data based on IoT hardware devices, and obtain corresponding model performance and structure data based on the training and performance optimization of the multimodal hybrid expert model.

[0038] The above data is preprocessed, such as data cleaning, to ensure the quality and consistency of the data. For example, natural language processing technology can be used to perform word segmentation and word vector representation on text data, that is, to segment the text into words so that the model can process them word by word, remove irrelevant characters, punctuation marks, numbers, etc., and retain meaningful text information, and then convert the text into a sequence form that the model can process; grayscale, resize, and adjust the channel order of image data, for example, the image size can be adjusted to a fixed size and the channel order can be adjusted to the format required by the model, so that the model can quickly receive data; for the adjustment of audio data, operations that can be performed include converting the audio sampling rate to the sampling rate required by the model and segmenting long audio into short segments of fixed length to ensure that the data can be effectively understood and processed by the model.

[0039] Construction and training of multimodal hybrid expert models. Before deploying the model, you need to build a complete multimodal hybrid expert model in advance. The model contains multiple expert sub-models, each of which is responsible for processing data of one or more modalities, such as DeepSeek-V3, Mixtral 8x7B, Qwen2.5-Max and other feasible models. One expert sub-model can specialize in processing text data, and another expert sub-model can process image data. Currently feasible hybrid expert models include: By introducing a hybrid expert mechanism, the flexibility and adaptability of the model can be improved.

[0040] Model and hardware node parameter setting. According to Table 1, the labels and parameters of each part of the trained multimodal hybrid expert model are summarized, and the relevant labels and parameters of the edge hardware resources are collected as the basis for the next step of model placement prediction; initialize the edge device relationship graph neural network and assign values ​​according to the type and characteristics of each node to generate multiple model nodes and hardware nodes, where the model nodes include data nodes, routing nodes, expert nodes, and output nodes; set the constraints of the hardware nodes based on the characteristic values ​​in each node; sort the expert nodes in a random and linear sequence manner, and use the heuristic enumeration method to combine and allocate the model nodes and hardware nodes to generate multiple candidate placement schemes for the hybrid expert model; judge the effectiveness of the candidate placement scheme based on the hardware constraints, filter out the infeasible candidate placement schemes, and select multiple feasible candidate placement schemes.

[0041] Construct a graph neural network and edge model placement prediction algorithm. Construct a graph neural network. According to the resource status of the edge device and the computing requirements of the model, use the edge model placement prediction algorithm in the present invention, take the edge device as the node in the graph, and the communication link between the devices as the edge. Capture the topological relationship and data dependency between the edge devices to obtain the optimal prediction solution, and reasonably distribute the different layers and expert models in the multimodal hybrid expert model to different edge devices. The algorithm will consider factors such as the computing power, storage capacity, and network bandwidth of the device, and make decisions based on the node characteristics of the graph neural network.

[0042] The feature embedding method is used to map the general feature data set into a continuous vector space to obtain a feature vector set; Based on the new message passing mechanism, according to the candidate placement scheme, the corresponding feature vector is transmitted from the expert node, data node, routing node and output node to the hardware node assigned to it; Then reversely transmit the characteristic information from the hardware node to the associated node; The feature information is passed in the order of the data flow in which the modal tokens are processed in each expert model run; In this way, edges are constructed according to the relationship between each node to obtain the edge device relationship graph neural network.

[0043] A cost function is established according to the resource status of the edge device and the computing requirements of the model. The edge device relationship graph neural network is iteratively trained to find candidate placement solutions with the goal of minimizing the cost function value, and the optimal placement solution of the hybrid expert model is obtained.

[0044] In summary, through the above specific implementation methods, the present invention can effectively solve the placement problem of the multimodal hybrid expert model in the edge environment, improve the running speed and load balance of the model in a resource-constrained environment, and ensure the successful deployment and stable operation of the model.

[0045] In this embodiment, a graph neural network (GNN) is applied to solve the problem of how to place a multimodal hybrid expert model in multiple weak resource edge devices under zero-sample conditions. A graph neural network is a deep learning model specifically used to process graph structured data. It consists of nodes and edges. Each point or edge contains feature data representing the entity it represents. A graph composed of multiple points and edges can represent the relationship and interaction between entities. By transmitting and aggregating information on graph structured data, the hidden information of nodes and edges in the graph structure is automatically transmitted and aggregated along the connected edges, thereby representing the mutual dependency by combining the connections between all adjacent nodes and edges of the node, and updating the global feature information of the node in the graph structure. Specifically including: General feature extraction and fusion: Taking into account edge hardware parameters and multimodal data, a data set containing general features (such as the number of CPUs, transmission bandwidth, etc.) is generated to help the model select the best placement solution under zero-sample conditions, providing a basis for subsequent model deployment and operation.

[0046] Heuristic solution generation: Generate multiple candidate placement solutions through heuristic enumeration, and judge the feasibility of the solutions based on hardware constraints, which broadens the scope of solution selection and ensures that the solutions can be implemented in the actual hardware environment, effectively improving the success rate of model placement and performance optimization level, so that the model can better adapt to the edge computing environment.

[0047] New message passing mechanism: The overall graph structure is constructed using feature data sets and candidate placement schemes. A new message passing mechanism is innovatively adopted to set the expert model and the corresponding edge device as independent nodes with feature attributes. Feature information is transmitted based on candidate schemes and data execution processes, so that the model can better capture the complex relationships between nodes and provide strong support for accurate prediction.

[0048] Embodiment 2 This embodiment provides an edge model scheduling system integrating graph neural network and hybrid expert model, including: A hybrid expert model generation module is configured to preprocess the multimodal data and generate a multimodal hybrid expert model based on the preprocessed multimodal data; A feature stitching module is configured to obtain edge device hardware parameters and multimodal data feature stitching to generate a general feature data set; A feasible candidate placement scheme determination module is configured to enumerate and generate multiple candidate placement schemes of a hybrid expert model based on a common feature data set using a heuristic algorithm, and screen out multiple feasible candidate placement schemes according to hardware constraints of edge devices; The graph neural network building module is configured to use a feature embedding method and a message passing mechanism to set the expert model and the corresponding edge device as independent nodes with feature attributes based on a common feature data set, and to establish edge connections according to feasible candidate placement solutions to build a graph neural network for edge device relationships; The optimal placement plan determination module is configured to establish a cost function based on the resource status of the edge device and the computing requirements of the model, iteratively train the edge device relationship graph neural network to obtain the candidate placement plan with the goal of minimizing the cost function value, and obtain the optimal placement plan of the hybrid expert model.

[0049] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0050] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0051] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0052] Embodiment 3 This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the edge model scheduling method of fusing graph neural and hybrid expert models as described in the first embodiment above are implemented.

[0053] Embodiment 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the edge model scheduling method of fusing graph neural and hybrid expert models as described in the first embodiment are implemented.

[0054] Embodiment 5 This embodiment provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the edge model scheduling method of the fusion graph neural network and hybrid expert model described in the first embodiment.

[0055] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0056] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0057] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0059] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0060] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. The edge model scheduling method integrating graph neural network and hybrid expert model is characterized by: include: Preprocess the multimodal data, and generate a multimodal hybrid expert model based on the preprocessed multimodal data; Obtain edge device hardware parameters and multimodal data feature splicing to generate a general feature data set; Based on the common feature dataset, a heuristic algorithm is used to enumerate and generate multiple candidate placement schemes of the hybrid expert model, and multiple feasible candidate placement schemes are screened out according to the hardware constraints of the edge device; Based on the common feature dataset, the expert model and the corresponding edge devices are set as independent nodes with feature attributes by using feature embedding method and message passing mechanism, and edge connections are established according to feasible candidate placement schemes to construct the edge device relationship graph neural network. A cost function is established according to the resource status of the edge device and the computing requirements of the model. The edge device relationship graph neural network is iteratively trained to find candidate placement solutions with the goal of minimizing the cost function value, and the optimal placement solution of the hybrid expert model is obtained.

2. The edge model scheduling method of fusion graph neural and hybrid expert model as claimed in claim 1, characterized in that: The processing process of the hybrid expert model is specifically as follows: Obtain multimodal data for standardization and feature extraction, and use feature embedding to map multimodal data into a unified feature space; Generate modal tokens corresponding to multiple modalities from the unified feature space to obtain multiple modal tokens; Using gated routing, the expert models are sorted in descending order of their weights, and multiple modal tokens are assigned to a corresponding number of expert models with high weight values ​​and not fully occupied for processing; The output results of multiple expert models are fused with the multiplication results of their weights and the original modal tokens to obtain a fusion result, which is then normalized and decoded as the final output result of the hybrid expert model.

3. The edge model scheduling method of fusion graph neural and hybrid expert model as claimed in claim 1, characterized in that: Based on the general feature data set, a heuristic algorithm is used to enumerate and generate multiple candidate placement schemes of the hybrid expert model, and multiple feasible candidate placement schemes are screened out according to the hardware constraints of the edge device, specifically: Initialize the edge device relationship graph neural network and assign values ​​according to the type and characteristics of each node to generate multiple model nodes and hardware nodes, where the model nodes include data nodes, routing nodes, expert nodes, and output nodes; Setting constraints on hardware nodes based on the eigenvalues ​​in each node; After sorting the expert nodes in random and linear sequence, the model nodes and hardware nodes are combined and allocated using a heuristic enumeration method to generate multiple candidate placement schemes for the hybrid expert model; The validity of candidate placement solutions is judged based on hardware constraints, infeasible candidate placement solutions are filtered out, and multiple feasible candidate placement solutions are screened out.

4. The edge model scheduling method of fusion graph neural and hybrid expert model as claimed in claim 3, characterized in that: The placement process of hardware nodes is combined and allocated with model nodes according to the following three rules: Each hardware node receives at least one model node; Hardware nodes are arranged from weak to strong based on resources and performance; During the placement process, model nodes are placed into hardware nodes in the order of data flow in a line.

5. The edge model scheduling method of fusion graph neural and hybrid expert model as claimed in claim 1, characterized in that: The construction process of the edge device relationship graph neural network is specifically as follows: The feature embedding method is used to map the general feature data set into a continuous vector space to obtain a feature vector set; According to the candidate placement scheme, the corresponding feature vector is transmitted from the expert node, the data node, the routing node and the output node to the hardware node assigned to it; Then reversely transmit the characteristic information from the hardware node to the associated node; The feature information is passed in the order of the data flow in which the modal tokens are processed in each expert model run; In this way, edges are constructed according to the relationship between each node to obtain the edge device relationship graph neural network.

6. The edge model scheduling method of fusion graph neural and hybrid expert model as claimed in claim 1, characterized in that: The cost function includes the amount of data processed by the expert model per unit time, the load balance of the edge nodes, and the communication time of the expert model in processing the modality token.

7. The edge model scheduling system integrating graph neural network and hybrid expert model is characterized by: include: A hybrid expert model generation module is configured to preprocess the multimodal data and generate a multimodal hybrid expert model based on the preprocessed multimodal data; A feature stitching module is configured to obtain edge device hardware parameters and multimodal data feature stitching to generate a general feature data set; A feasible candidate placement scheme determination module is configured to enumerate and generate multiple candidate placement schemes of a hybrid expert model based on a common feature data set using a heuristic algorithm, and screen out multiple feasible candidate placement schemes according to hardware constraints of edge devices; The graph neural network building module is configured to use a feature embedding method and a message passing mechanism to set the expert model and the corresponding edge device as independent nodes with feature attributes based on a common feature data set, and to establish edge connections according to feasible candidate placement solutions to build a graph neural network for edge device relationships; The optimal placement plan determination module is configured to establish a cost function based on the resource status of the edge device and the computing requirements of the model, iteratively train the edge device relationship graph neural network to obtain the candidate placement plan with the goal of minimizing the cost function value, and obtain the optimal placement plan of the hybrid expert model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the edge model scheduling method of the fusion graph neural and hybrid expert model as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the edge model scheduling method of fusing graph neural and hybrid expert models as described in any one of claims 1-6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in the edge model scheduling method of fusing graph neural and hybrid expert models according to any one of claims 1 to 6 are implemented.

Images