Model reasoning method and device, electronic equipment and computer program product
By fragmenting large-scale language models and distributing them to edge devices for inference tasks, combined with dynamic programming and real-time monitoring, the problems of high latency and uneven resource utilization in cloud computing are solved, achieving efficient model inference and real-time requirements, and is suitable for smart terminals and IoT scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA TELECOM CORP LTD TECHNOLOGY INNOVATION CENTER
- Filing Date
- 2024-10-24
- Publication Date
- 2026-04-24
AI Technical Summary
Traditional centralized cloud computing methods are insufficient to meet the real-time reasoning needs of large-scale language models in smart terminal devices and IoT scenarios. They face problems such as bandwidth limitations, high latency, and uneven resource utilization, and lack effective distributed collaborative processing capabilities.
By splitting the target model into multiple hierarchical models and distributing them to various edge devices for inference tasks, and combining dynamic programming and real-time monitoring, the scheduling strategy is optimized to minimize the total inference time and resource utilization.
It achieves efficient model inference under the cloud-edge collaborative architecture, reduces bandwidth consumption and latency, improves system performance and resource utilization, adapts to complex edge computing scenarios, and meets real-time and privacy protection requirements.
Smart Images
Figure CN121920506A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a model reasoning method, a model reasoning device, an electronic device, and a computer program product. Background Technology
[0002] With the widespread application of large-scale language models, the demand for data processing and reasoning has grown dramatically, especially in scenarios such as smart terminal devices, the Internet of Things, and the Industrial Internet. Real-time reasoning, low latency, and high efficiency have become crucial requirements.
[0003] However, traditional centralized cloud computing inference methods struggle to meet these needs, primarily facing issues such as bandwidth limitations, high latency, and uneven resource utilization. The high computational demands and complexity of large-model inference render traditional inference methods inadequate for handling latency-sensitive tasks. Furthermore, large-model inference typically lacks effective distributed collaborative processing capabilities, which is particularly evident in edge computing scenarios.
[0004] Therefore, there is an urgent need in this field for a model inference method that can realize the allocation and optimized scheduling of large model inference tasks under the cloud-edge collaborative architecture, and can effectively improve the efficiency of model inference.
[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0006] The purpose of this disclosure is to provide a model inference method, model inference device, electronic device and computer program product, which can at least to some extent realize the allocation and optimized scheduling of large model inference tasks under the cloud-edge collaborative architecture, and effectively improve the efficiency of model inference.
[0007] According to a first aspect of this disclosure, a model inference method is provided, applied to a cloud server, comprising:
[0008] The target model is segmented to obtain multiple hierarchical models of the target model, and each hierarchical model corresponds to a reasoning task at a different stage in the target model.
[0009] The hierarchical model is assigned to each edge device, so that each edge device performs a corresponding inference task based on the hierarchical model and obtains the inference result corresponding to the hierarchical model.
[0010] Receive the complete inference result returned by the edge device, which is based on the inference results corresponding to each of the hierarchical models.
[0011] In one exemplary embodiment of this disclosure, the step of segmenting the target model includes:
[0012] The resource status parameters of each edge device are obtained, and the target model is segmented according to the number of edge devices and the resource status parameters of each edge device.
[0013] In one exemplary embodiment of this disclosure, the step of assigning the hierarchical model to each edge device includes:
[0014] The total inference time of all the hierarchical models is obtained based on the inference task computation time of each of the hierarchical models on each of the edge devices, and the communication time between every two adjacent hierarchical models, wherein the inference task computation time and the communication time are determined by the resource status parameters of the edge devices.
[0015] By minimizing the total inference time, a scheduling strategy for the hierarchical model is obtained, and the hierarchical model is allocated to each edge device based on the scheduling strategy.
[0016] In one exemplary embodiment of this disclosure, obtaining the scheduling strategy of the hierarchical model by minimizing the total inference time includes:
[0017] Based on the minimum cumulative inference time corresponding to the allocation of the hierarchical model to the edge device, the state transition equation of the hierarchical model is established;
[0018] The minimum total inference time is determined based on the state transition equation, and the scheduling strategy of the hierarchical model is determined when the total inference time is at its minimum.
[0019] In one exemplary embodiment of this disclosure, the method further includes:
[0020] The resource status parameters of each edge device are monitored in real time, and the scheduling strategy of the hierarchical model is adjusted based on the changes in the resource status parameters.
[0021] In one exemplary embodiment of this disclosure, the method further includes:
[0022] The input data of the target model is divided into multiple batches of fragmented input data, and the fragmented input data is distributed to each edge device so that the edge device can perform corresponding inference tasks based on the fragmented input data.
[0023] In one exemplary embodiment of this disclosure, the method further includes:
[0024] The task processing status of each edge device is obtained in real time.
[0025] In one exemplary embodiment of this disclosure, the method further includes:
[0026] The resource status parameters of each edge device are monitored in real time, and a target transmission path is determined based on the resource status parameters of the edge devices, so that the edge devices can return the complete inference result through the target transmission path.
[0027] According to a second aspect of this disclosure, a model inference method is provided, applied to an edge device, comprising:
[0028] The system receives a hierarchical model allocated by the cloud server, and the different hierarchical models correspond to different stages of the inference task in the target model.
[0029] Based on the hierarchical model, the corresponding reasoning task is executed to obtain the reasoning result corresponding to the hierarchical model;
[0030] The inference results of each of the hierarchical models are aggregated to obtain a complete inference result, and the complete inference result is returned to the cloud server.
[0031] In one exemplary embodiment of this disclosure, the aggregation of the inference results of each of the hierarchical models includes:
[0032] The inference results of each hierarchical model are aggregated based on a preset aggregation function.
[0033] According to a third aspect of this disclosure, a model inference apparatus is provided, applied to a cloud server, comprising:
[0034] The target model sharding module is used to shard the target model to obtain multiple hierarchical models of the target model, and each hierarchical model corresponds to a reasoning task at a different stage in the target model.
[0035] The inference task allocation module is used to allocate the hierarchical model to each edge device, so that each edge device can perform the corresponding inference task based on the hierarchical model and obtain the inference result corresponding to the hierarchical model.
[0036] The inference result receiving module is used to receive the complete inference result returned by the edge device based on the inference result corresponding to each of the hierarchical models.
[0037] According to a fourth aspect of this disclosure, a model inference apparatus is provided for use in an edge device, comprising:
[0038] The inference task receiving module is used to receive the hierarchical model allocated by the cloud server. Different hierarchical models correspond to different stages of inference tasks in the target model.
[0039] The reasoning task execution module is used to execute the corresponding reasoning task based on the hierarchical model and obtain the reasoning result corresponding to the hierarchical model.
[0040] The inference result return module is used to aggregate the inference results of each of the hierarchical models to obtain a complete inference result after aggregation, and return the complete inference result to the cloud server.
[0041] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the model inference method described in any of the preceding claims by executing the executable instructions.
[0042] According to a sixth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the model reasoning method described in any of the preceding claims.
[0043] The exemplary embodiments disclosed herein can have the following beneficial effects:
[0044] In the model inference method of this exemplary embodiment, the cloud server obtains multiple hierarchical models of the target model through model sharding. Each hierarchical model corresponds to an inference task at a different stage in the target model. The hierarchical models are then distributed to various edge devices, enabling each edge device to execute its corresponding inference task based on the hierarchical model and obtain the inference result corresponding to the hierarchical model. Finally, the cloud server receives the complete inference result returned by the edge devices, based on the inference results corresponding to each hierarchical model. The edge devices receive the hierarchical models distributed by the cloud server, execute their corresponding inference tasks based on the hierarchical models, obtain the inference results corresponding to the hierarchical models, aggregate the inference results of each hierarchical model to obtain the aggregated complete inference result, and return the complete inference result to the cloud server.
[0045] The model inference method in the exemplary embodiments of this disclosure completes model inference through collaboration between cloud servers and edge devices. The cloud server handles large-scale computing tasks, while the edge devices are responsible for latency-sensitive inference tasks. On the one hand, it can fully utilize the computing power and memory resources of heterogeneous edge devices, effectively share the computing burden, significantly improve inference throughput, achieve efficient inference of large-scale models, reduce bandwidth consumption and latency, and improve overall system performance and resource utilization. On the other hand, it realizes the allocation and optimized scheduling of large-scale model inference tasks under the cloud-edge collaborative architecture, enabling it to adapt to complex edge computing scenarios. It achieves efficient deployment and inference of large-scale models in environments with limited edge computing resources, meeting the higher demands of future intelligent applications for real-time performance, efficiency, and privacy protection.
[0046] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0048] Figure 1 A flowchart illustrating a model inference method applied to a cloud server according to an exemplary embodiment of this disclosure is shown.
[0049] Figure 2 A flowchart illustrating a model inference method applied to an edge device according to an exemplary embodiment of this disclosure is shown.
[0050] Figure 3 The illustration shows a schematic diagram of model sharding and cloud-edge collaborative task scheduling according to a specific embodiment of the present disclosure;
[0051] Figure 4 A schematic diagram illustrating the process of assigning a hierarchical model to various edge devices according to an exemplary embodiment of this disclosure is shown.
[0052] Figure 5 A flowchart illustrating the determination of a scheduling strategy using a dynamic programming method in an exemplary embodiment of this disclosure is shown.
[0053] Figure 6 A flowchart illustrating a cloud-edge collaborative model inference method according to a specific embodiment of the present disclosure is shown.
[0054] Figure 7 A block diagram of a model inference apparatus applied to a cloud server according to an exemplary embodiment of the present disclosure is shown;
[0055] Figure 8 A block diagram of a model inference apparatus applied to an edge device according to an exemplary embodiment of the present disclosure is shown;
[0056] Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown. Detailed Implementation
[0057] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.
[0058] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0059] Large-scale model inference refers to the process of using large-scale language models to process input data and generate output results. Because large models contain billions to hundreds of billions of parameters, the inference process usually requires a large amount of computing resources and memory.
[0060] With the widespread adoption of large-scale language models (LLMs), large-model inference methods face significant challenges, especially when dealing with complex computational tasks. First, large models encounter high latency and bandwidth bottlenecks during cloud-based inference, severely impacting real-time performance requirements. Second, edge computing devices have limited resources and cannot directly support and run large-scale models.
[0061] In some related embodiments, cloud-edge collaborative computing can be used for large model inference. Cloud-edge collaborative computing refers to the collaboration between cloud servers and edge devices to share computing tasks and achieve optimal resource utilization. While cloud-edge collaborative inference can partially solve this problem, static rule-based model sharding and task scheduling strategies are difficult to fully utilize heterogeneous computing resources, resulting in low inference efficiency and unstable system performance. How to reasonably allocate inference tasks between the cloud and edge devices and maximize resource utilization while ensuring real-time performance remains a challenge. Relying solely on static rules or fixed strategies makes it difficult to adapt to dynamic environmental changes. As large model applications further develop, how to achieve dynamic model sharding, task scheduling, parallel inference, and rapid transmission and aggregation of inference results remains a pressing challenge in large model inference technology.
[0062] To address the aforementioned issues, this example implementation first provides a model inference method applied to a cloud server. (Reference) Figure 1 As shown, the above model inference method may include the following steps:
[0063] Step S110. The target model is segmented to obtain multiple hierarchical models of the target model, each of which corresponds to a different stage of the reasoning task in the target model.
[0064] Step S120. Distribute the hierarchical model to each edge device so that each edge device can perform the corresponding inference task based on the hierarchical model and obtain the inference result corresponding to the hierarchical model.
[0065] Step S130. Receive the complete inference result returned by the edge device based on the inference results corresponding to each layered model.
[0066] This example implementation also provides a model inference method applied to edge devices. (Reference) Figure 2 As shown, the above model inference method may include the following steps:
[0067] Step S210. Receive the layered model allocated by the cloud server. Different layered models correspond to different stages of inference tasks in the target model.
[0068] Step S220. Execute the corresponding inference task based on the hierarchical model to obtain the inference result corresponding to the hierarchical model.
[0069] Step S230. Aggregate the inference results of each hierarchical model to obtain the aggregated complete inference result, and return the complete inference result to the cloud server.
[0070] The model inference method in the exemplary embodiments of this disclosure completes model inference through collaboration between cloud servers and edge devices. The cloud server handles large-scale computing tasks, while the edge devices are responsible for latency-sensitive inference tasks. On the one hand, it can fully utilize the computing power and memory resources of heterogeneous edge devices, effectively share the computing burden, significantly improve inference throughput, achieve efficient inference of large-scale models, reduce bandwidth consumption and latency, and improve overall system performance and resource utilization. On the other hand, it realizes the allocation and optimized scheduling of large-scale model inference tasks under the cloud-edge collaborative architecture, enabling it to adapt to complex edge computing scenarios. It achieves efficient deployment and inference of large-scale models in environments with limited edge computing resources, meeting the higher demands of future intelligent applications for real-time performance, efficiency, and privacy protection.
[0071] The model inference method in the exemplary embodiments of this disclosure is applicable to future edge-side large-scale model application scenarios requiring real-time response, such as smartphones, smart homes, and the Industrial Internet of Things (IIoT). For example, on terminal devices such as smartphones and smart speakers, cloud-edge collaborative large-scale model inference can provide more real-time natural language processing services, such as voice assistants, translation, and personalized recommendations, improving user experience. In scenarios such as smart manufacturing and smart logistics, edge computing is used to process large-scale model inference tasks, reducing latency and enabling real-time data analysis and decision-making between devices. Through efficient inference of large models on edge devices, the linkage of smart home devices and real-time response in smart city management can be achieved, supporting future smart living. This method has good scalability, can be flexibly expanded as the network scale grows, and is widely applicable to edge computing environments of different sizes.
[0072] Below, in conjunction with Figures 3 to 5 The steps described above in this example implementation will be explained in more detail.
[0073] In step S110, the target model is segmented to obtain multiple hierarchical models of the target model, each of which corresponds to a different stage of the reasoning task in the target model.
[0074] In this example implementation, the target model is a large-scale language model, and its inference process refers to the process of using the large-scale language model to process the input data and generate output results. Model fragmentation involves splitting the large-scale language model into multiple fragments and allocating different fragments to different edge devices based on parameters such as device resources and network conditions, for executing corresponding inference tasks at different stages.
[0075] like Figure 3 The diagram illustrates model sharding and cloud-edge collaborative task scheduling. By optimizing algorithms to rationally allocate large-scale language models to heterogeneous edge devices, system resource utilization can be maximized and inference efficiency improved.
[0076] In this example implementation, the cloud server can obtain the resource status parameters of each edge device and perform sharding processing on the target model based on the number of edge devices and the resource status parameters of each edge device.
[0077] Assuming the target model has N layers, including a data processing layer, an intermediate layer, an output layer, etc., the system can partition the model based on the resource status parameters of the edge devices, including device memory and computing power, and define a binary variable X. i,j , indicating that the i-th layer is assigned to device j. The constraints are:
[0078]
[0079] Where M is the number of edge devices, and each layer of the model is assigned to one edge device.
[0080] In step S120, the hierarchical model is assigned to each edge device so that each edge device can perform the corresponding inference task based on the hierarchical model and obtain the inference result corresponding to the hierarchical model.
[0081] The task scheduling module on the cloud server can dynamically select edge devices to execute inference tasks based on information such as the computing power and network bandwidth of the edge devices, with the optimization goal of minimizing the total inference time.
[0082] In this example implementation, such as Figure 4 As shown, assigning the hierarchical model to various edge devices can specifically include the following steps:
[0083] Step S410. Based on the inference task computation time of each hierarchical model on each edge device and the communication time between every two adjacent hierarchical models, obtain the total inference time of all hierarchical models.
[0084] The computation and communication times for inference tasks are determined by the resource status parameters of the edge devices.
[0085] Total reasoning time T total It is the cumulative sum of computation time and communication time, calculated using the following formula:
[0086]
[0087] in, This represents the computation time of the i-th layer on device j, which varies depending on the processing power of the device. This represents the communication time for transmitting data from device k to device j. That is, when the (i-1)th layer and the ith layer of the model are assigned to different devices, the computation result of the (i-1)th layer needs to be transmitted from device k to device j. Communication time is affected by factors such as network bandwidth and data size. If both layers are on the same device, the communication time is zero. In the above T... total In the calculation formula, the first part The computation time for each layer of the model was calculated on each device. (Part Two) The communication time between adjacent layers was then calculated.
[0088] Step S420. By minimizing the total inference time, the scheduling strategy of the hierarchical model is obtained, and the hierarchical model is allocated to each edge device based on the scheduling strategy of the hierarchical model.
[0089] In this example implementation, dynamic programming can be used to minimize the total inference time. For example... Figure 5 As shown, by minimizing the total inference time, the scheduling strategy of the hierarchical model is obtained, which can specifically include the following steps:
[0090] Step S510. Establish the state transition equation of the hierarchical model based on the minimum cumulative inference time corresponding to the allocation of the hierarchical model to the edge device.
[0091] Based on dynamic programming, we define the state DP(i,j) as the minimum cumulative inference time when the i-th layer is assigned to edge device j. The state transition equation considers computation and communication time. The recursive formula is:
[0092]
[0093] That is, the time when the i-th layer is allocated to device j is determined by the cumulative time when the i-1-th layer is allocated to device k plus the calculation and communication time.
[0094] The initial condition for the above recursive formula is to initialize the inference time of layer 0 on each device.
[0095] Step S520. Determine the minimum total inference time based on the state transition equation, and determine the scheduling strategy of the hierarchical model when the total inference time is at its minimum.
[0096] By traversing all possible device allocation paths, we find the path that minimizes the total inference time T. total The optimal allocation strategy can be found by using a dynamic programming table to find the minimum value of the last level and backtracking to derive the optimal allocation strategy.
[0097] In this example implementation, by employing an optimized model sharding and device selection algorithm, based on the device's computing power, memory capacity, and network bandwidth, dynamic programming and optimization algorithms are used to shard the large model and allocate it to suitable devices. This intelligently allocates different parts of the model, ensuring optimal configuration of inference efficiency and system response speed. It maximizes resource utilization and reduces system latency, thus realizing the allocation and optimized scheduling of large model inference tasks under a cloud-edge collaborative architecture.
[0098] In this example implementation, the cloud server can also monitor the resource status parameters of each edge device in real time and adjust the scheduling strategy of the hierarchical model based on the changes in the resource status parameters.
[0099] The task scheduling module in the cloud server can monitor device resources and network conditions in real time, and continuously adjust device selection and model sharding strategies based on dynamic programming algorithms to ensure optimal configuration of inference performance. This strategy makes full use of edge device resources and maximizes inference efficiency through optimized scheduling. By monitoring the device status in real time, the system can dynamically adjust task allocation and scheduling strategies to ensure stability and efficiency in complex network environments.
[0100] In this example implementation, the cloud server can also divide the input data of the target model into multiple batches of fragmented input data and distribute the fragmented input data to various edge devices so that the edge devices can perform corresponding inference tasks based on the fragmented input data.
[0101] The task scheduling module in a cloud server can divide input data into multiple small batches, with each batch processed in parallel on different edge devices. Assume each device j processes batch b. i The time is Total time T pipeline It can be calculated using the following formula:
[0102]
[0103] Where B represents the total number of batches, and the max operation represents finding the device time that processes the longest batch.
[0104] In this example implementation, by introducing a pipelined parallel inference task scheduling strategy, the inference task is divided into multiple small batches and the inference process is executed in parallel. Pipeline parallel processing is used to improve the system throughput and reduce latency, thereby achieving efficient parallel inference and further improving the overall inference efficiency.
[0105] In this example implementation, the cloud server can also obtain the task processing status of each edge device in real time.
[0106] The task scheduling module in the cloud server synchronizes tasks across devices to ensure that all batches can be aggregated in a timely manner after parallel processing. Simultaneously, by optimizing scheduling strategies, the module can dynamically allocate tasks, preventing certain devices from becoming performance bottlenecks and ensuring smooth pipeline operation.
[0107] In step S130, the complete inference result is received from the edge device, which is based on the inference results corresponding to each hierarchical model.
[0108] Once the edge device has completed all inference tasks, it will return the aggregated inference results to the cloud server to obtain the complete inference result of the target model.
[0109] In this example implementation, the cloud server can monitor the resource status parameters of each edge device in real time and determine the target transmission path based on the resource status parameters of the edge devices, so that the edge devices can return complete inference results through the target transmission path.
[0110] The task scheduling module in the cloud server can dynamically select the optimal path to transmit the aggregated results back to the cloud server or directly to the user device based on current network bandwidth and latency conditions. The optimization objective is to minimize the transmission time T. trans The calculation formula is as follows:
[0111]
[0112] Where Data_size is the data size, Bandwidth is the bandwidth, and Network_delay is the network latency.
[0113] The task scheduling module can adjust the transmission path and strategy in real time according to network conditions to ensure the timeliness and accuracy of the results.
[0114] In step S210, the layered models allocated by the cloud server are received, and the different layered models correspond to the inference tasks at different stages in the target model.
[0115] Edge devices receive a hierarchical model from the cloud server and acquire inference tasks at different stages of the target model.
[0116] In step S220, the corresponding inference task is performed based on the hierarchical model to obtain the inference result corresponding to the hierarchical model.
[0117] Edge devices execute corresponding inference tasks based on the hierarchical model, obtaining inference results for each layer of the model.
[0118] In step S230, the inference results of each hierarchical model are aggregated to obtain the aggregated complete inference result, and the complete inference result is returned to the cloud server.
[0119] In this example implementation, the inference results of each hierarchical model can be aggregated based on a preset aggregation function.
[0120] After multiple edge devices complete different inference tasks in parallel, the system dynamically aggregates the dispersed inference results. Let R be the inference result of each device j. j The system uses an algorithm to merge these results to generate the complete output R. final :
[0121] R final =Aggregate(R1,R2,...,R) M )
[0122] Here, Aggregate represents an aggregation function, which can integrate results in different forms according to task requirements, such as concatenation, weighted averaging, etc.
[0123] After aggregating the inference results of each hierarchical model, the complete inference results can be returned to the cloud server through the target transmission path determined by the cloud server.
[0124] In this example implementation, by combining real-time network conditions, dynamically aggregating inference results, and selecting the optimal transmission path, the system ensures rapid transmission of inference results and overall system responsiveness, thereby improving system throughput and efficiency.
[0125] like Figure 6 The diagram shown is a flowchart illustrating a cloud-edge collaborative model inference method according to a specific embodiment of this disclosure. It serves as an example of the steps described above in this exemplary embodiment. The specific steps of the flowchart are as follows:
[0126] Step S610. Construction of the cloud-edge collaborative system.
[0127] Build a collaborative system between cloud servers and edge devices, where cloud servers handle large-scale computing tasks and edge devices are responsible for latency-sensitive inference tasks, supporting dynamic resource allocation and optimization.
[0128] Step S620. Model partitioning and task scheduling.
[0129] The system segments a large-scale language model and, through optimization algorithms, allocates the segmented tasks to appropriate devices based on factors such as device resources and network bandwidth. The task scheduling module runs on a cloud server, monitoring device and network status in real time and dynamically adjusting task allocation to ensure optimal inference configuration.
[0130] Step S630. Model fragment reception and inference task execution.
[0131] Edge devices receive model fragments and execute inference tasks according to scheduling strategies. Different devices handle different stages or levels of model inference.
[0132] Step S640. Pipeline parallel inference.
[0133] To improve efficiency, the system employs pipelined parallel inference, dividing tasks into multiple small batches for parallel processing, reducing latency and increasing throughput.
[0134] Step S650. Aggregate and return the inference results.
[0135] After the edge device completes the inference task, the system aggregates the scattered results to generate a complete output. The aggregated result is returned to the cloud server or directly delivered to the user device to ensure timely response.
[0136] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0137] Furthermore, this disclosure also provides a model inference apparatus for use on a cloud server. (Reference) Figure 7 As shown, the model inference device may include a target model fragmentation module 710, an inference task allocation module 720, and an inference result receiving module 730. Wherein:
[0138] The target model sharding module 710 can be used to shard the target model to obtain multiple hierarchical models of the target model, each hierarchical model corresponding to the reasoning task at a different stage in the target model;
[0139] The inference task allocation module 720 can be used to allocate the hierarchical model to each edge device, so that each edge device can execute the corresponding inference task based on the hierarchical model and obtain the inference result corresponding to the hierarchical model.
[0140] The inference result receiving module 730 can be used to receive the complete inference result returned by the edge device based on the inference result corresponding to each layer model.
[0141] In some exemplary embodiments of this disclosure, the target model sharding module 710 may include a resource status determination unit, which can be used to obtain resource status parameters of each edge device and shard the target model according to the number of edge devices and the resource status parameters of each edge device.
[0142] In some exemplary embodiments of this disclosure, the inference task allocation module 720 may include a total inference time determination unit and a hierarchical model allocation unit. Wherein:
[0143] The total inference time determination unit can be used to obtain the total inference time of all layered models based on the inference task computation time of each layered model on each edge device and the communication time between every two adjacent layered models. The inference task computation time and communication time are determined by the resource status parameters of the edge devices.
[0144] The hierarchical model allocation unit can be used to obtain the scheduling strategy of the hierarchical model by minimizing the total inference time, and to allocate the hierarchical model to each edge device based on the scheduling strategy of the hierarchical model.
[0145] In some exemplary embodiments of this disclosure, the hierarchical model allocation unit may include a state transition equation establishment unit and a scheduling strategy determination unit. Wherein:
[0146] The state transition equation establishment unit can be used to establish the state transition equation of the hierarchical model based on the minimum cumulative inference time corresponding to the allocation of the hierarchical model to the edge device.
[0147] The scheduling strategy determination unit can be used to determine the minimum total inference time based on the state transition equation, and determine the scheduling strategy of the hierarchical model when the total inference time is at its minimum.
[0148] In some exemplary embodiments of this disclosure, the inference task allocation module 720 may further include a scheduling strategy adjustment unit, which can be used to monitor the resource status parameters of each edge device in real time and adjust the scheduling strategy of the hierarchical model based on the changes in the resource status parameters.
[0149] In some exemplary embodiments of this disclosure, a model inference apparatus provided by this disclosure may further include an input data sharding module, which can be used to divide the input data of the target model into multiple batches of sharded input data and distribute the sharded input data to various edge devices so that the edge devices can perform corresponding inference tasks based on the sharded input data.
[0150] In some exemplary embodiments of this disclosure, a model inference apparatus provided by this disclosure may further include a task processing status monitoring module, which can be used to acquire the task processing status of each edge device in real time.
[0151] In some exemplary embodiments of this disclosure, the model inference apparatus provided by this disclosure may further include a target transmission path determination module, which can be used to monitor the resource status parameters of each edge device in real time and determine the target transmission path based on the resource status parameters of the edge device, so that the edge device returns the complete inference result through the target transmission path.
[0152] Furthermore, this disclosure also provides a model inference apparatus for use in edge devices. (Reference) Figure 8 As shown, the model inference device may include an inference task receiving module 810, an inference task execution module 820, and an inference result return module 830. Wherein:
[0153] The inference task receiving module 810 can be used to receive the hierarchical model allocated by the cloud server. Different hierarchical models correspond to different stages of inference tasks in the target model.
[0154] The inference task execution module 820 can be used to execute the corresponding inference task based on the hierarchical model and obtain the inference result corresponding to the hierarchical model.
[0155] The inference result return module 830 can be used to aggregate the inference results of each hierarchical model to obtain the aggregated complete inference result, and return the complete inference result to the cloud server.
[0156] In some exemplary embodiments of this disclosure, the inference result return module 830 may include an inference result aggregation unit, which can be used to aggregate the inference results of each hierarchical model based on a preset aggregation function.
[0157] The specific details of each module / unit in the above-mentioned model inference device have been described in detail in the corresponding method implementation section, and will not be repeated here.
[0158] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0159] Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.
[0160] It should be noted that, Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0161] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0162] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.
[0163] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs various functions defined in the system of this disclosure.
[0164] Exemplary embodiments of this disclosure also provide a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the model inference method described above.
[0165] In one embodiment, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing the computer program, such as read-only memory, NAND flash memory, etc.
[0166] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.
[0167] Computer program code can be written in one or more programming languages. Examples of programming languages include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).
[0168] Computer programs can be carried or transmitted via signals such as electricity, magnetism, light, electromagnetic fields, and infrared radiation. Electronic devices can convert the signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to execute) the method steps of various exemplary embodiments of this disclosure, such as the model inference method described above.
[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0170] It should be noted that although several modules for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.
[0171] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.
[0172] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A model inference method applied to a cloud server, characterized in that, include: The target model is segmented to obtain multiple hierarchical models of the target model, and each hierarchical model corresponds to a reasoning task at a different stage in the target model. The hierarchical model is assigned to each edge device, so that each edge device performs a corresponding inference task based on the hierarchical model and obtains the inference result corresponding to the hierarchical model. Receive the complete inference result returned by the edge device, which is based on the inference results corresponding to each of the hierarchical models.
2. The model reasoning method according to claim 1, characterized in that, The step of segmenting the target model includes: The resource status parameters of each edge device are obtained, and the target model is segmented according to the number of edge devices and the resource status parameters of each edge device.
3. The model reasoning method according to claim 1, characterized in that, The process of assigning the hierarchical model to each edge device includes: The total inference time of all the hierarchical models is obtained based on the inference task computation time of each of the hierarchical models on each of the edge devices and the communication time between every two adjacent hierarchical models, wherein the inference task computation time and the communication time are determined by the resource status parameters of the edge devices. By minimizing the total inference time, a scheduling strategy for the hierarchical model is obtained, and the hierarchical model is allocated to each edge device based on the scheduling strategy.
4. The model reasoning method according to claim 3, characterized in that, The method of obtaining the scheduling strategy for the hierarchical model by minimizing the total inference time includes: Based on the minimum cumulative inference time corresponding to the allocation of the hierarchical model to the edge device, the state transition equation of the hierarchical model is established; The minimum total inference time is determined based on the state transition equation, and the scheduling strategy of the hierarchical model is determined when the total inference time is at its minimum.
5. The model reasoning method according to claim 3, characterized in that, The method further includes: The resource status parameters of each edge device are monitored in real time, and the scheduling strategy of the hierarchical model is adjusted based on the changes in the resource status parameters.
6. The model reasoning method according to claim 1, characterized in that, The method further includes: The input data of the target model is divided into multiple batches of fragmented input data, and the fragmented input data is distributed to each edge device so that the edge device can perform corresponding inference tasks based on the fragmented input data.
7. The model reasoning method according to claim 1, characterized in that, The method further includes: The task processing status of each edge device is obtained in real time.
8. The model reasoning method according to claim 1, characterized in that, The method further includes: The resource status parameters of each edge device are monitored in real time, and a target transmission path is determined based on the resource status parameters of the edge devices, so that the edge devices return the complete inference result through the target transmission path.
9. A model inference method applied to edge devices, characterized in that, include: The system receives a hierarchical model allocated by the cloud server, and the different hierarchical models correspond to different stages of the inference task in the target model. Based on the hierarchical model, the corresponding reasoning task is executed to obtain the reasoning result corresponding to the hierarchical model; The inference results of each of the hierarchical models are aggregated to obtain a complete inference result, and the complete inference result is returned to the cloud server.
10. The model reasoning method according to claim 9, characterized in that, The aggregation of the inference results of each of the hierarchical models includes: The inference results of each hierarchical model are aggregated based on a preset aggregation function.
11. A model inference device, applied to a cloud server, characterized in that, include: The target model sharding module is used to shard the target model to obtain multiple hierarchical models of the target model, each of which corresponds to a reasoning task at a different stage in the target model. The inference task allocation module is used to allocate the hierarchical model to each edge device, so that each edge device can perform the corresponding inference task based on the hierarchical model and obtain the inference result corresponding to the hierarchical model. The inference result receiving module is used to receive the complete inference result returned by the edge device based on the inference result corresponding to each of the hierarchical models.
12. A model inference device applied to an edge device, characterized in that, include: The inference task receiving module is used to receive the hierarchical model allocated by the cloud server. Different hierarchical models correspond to different stages of inference tasks in the target model. The reasoning task execution module is used to execute the corresponding reasoning task based on the hierarchical model and obtain the reasoning result corresponding to the hierarchical model. The inference result return module is used to aggregate the inference results of each of the hierarchical models to obtain a complete inference result after aggregation, and return the complete inference result to the cloud server.
13. An electronic device, characterized in that, include: processor; as well as A memory for storing one or more programs, which, when executed by the processor, cause the processor to implement the model inference method as described in any one of claims 1 to 10.
14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the model reasoning method as described in any one of claims 1 to 10.