Early exit based multi-modal scheduling method for edge-constrained devices
By establishing a multimodal scheduling framework on edge-constrained devices and utilizing early exit mechanisms and Lyapunov optimization to generate modal offloading strategies, the problems of high computational cost and complex scheduling of multimodal models on edge devices are solved, thereby improving processing speed while ensuring accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHEN ZHEN WAN ZHI DA QI YE GUAN LI YOU XIAN GONG SI
- Filing Date
- 2024-08-23
- Publication Date
- 2026-07-24
AI Technical Summary
Deploying multimodal models on edge-constrained devices involves large computational loads and complex scheduling, making it difficult to improve running speed while ensuring model accuracy.
An early exit mechanism and Lyapunov optimization are adopted to establish a multimodal scheduling framework. Through collaborative processing between edge devices and the server, a modal offloading strategy is generated. Multimodal offloading scheduling is performed by minimizing the optimization objective using early exit nodes and Lyapunov drift penalties.
Reduce the computational overhead of edge devices, ensure system stability and model processing accuracy, and improve system processing speed.
Smart Images
Figure CN119127335B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of resource offloading technology, and more specifically, to a multimodal scheduling method based on early exit for edge-constrained devices. Background Technology
[0002] Multimodal collaborative processing involves the model acquiring data from multiple modalities and then working together to obtain the final result. Compared with traditional single-modal processing, multimodal processing can improve the accuracy of the model. For example, in multispectral object recognition, infrared data and RGB data are used to improve the success rate of model detection.
[0003] Due to factors such as cost, most models need to be deployed on edge devices with limited computing power (referred to as "edge-constrained devices"). While multimodal models improve model accuracy, they also increase computational load. Therefore, it is crucial to rationally schedule multimodal models to improve running speed on edge devices while maintaining a certain level of accuracy. Many past studies have focused on offloading tasks from edge devices to servers to improve their processing speed. Dynamic programming was commonly used to find the optimal solution by pruning the entire search space. However, when model accuracy and the overall environment change, the scheduling problem becomes NP-hard, making it difficult to find the optimal solution within a limited time. Some researchers have begun using ant colony optimization and simulated annealing to schedule resource offloading from edge devices, but these methods introduce significant uncertainty and fail to stabilize the system. Recently, reinforcement learning has been proposed to offload resources from edge devices for unimodal models. Reinforcement learning is used to schedule edge device resources to determine whether to offload them, but it also introduces additional computational overhead.
[0004] Furthermore, previous computational task offloading scheduling algorithms have all used single-modality models as experimental subjects. In multimodal models, the combination of different modes will bring different processing results, which is far more complex than the multimodal case. Therefore, the scheduling algorithm needs to dynamically schedule the input of each mode according to the environmental conditions, so as to ensure the accuracy of the model and improve the system processing speed. Summary of the Invention
[0005] To overcome the drawback of high computational overhead when deploying multimodal models on edge devices as described in the prior art, this invention provides a multimodal scheduling method based on early exit for edge-constrained devices.
[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:
[0007] Firstly, a multimodal scheduling method based on early exit for edge-constrained devices, applied to the field of multispectral object recognition, includes:
[0008] A multimodal scheduling framework is established, including an edge device and a server. The edge device is equipped with a first modal model for processing first spectral data to obtain a first modal recognition result, and the server is equipped with a second modal model for processing second spectral data to obtain a second modal recognition result.
[0009] The early exit mechanism is used to obtain several early exit nodes for the first modal model;
[0010] Under the maximum delay constraint, based on network environment information and the early exit node, a multimodal offloading scheduling system model is established based on Lyapunov optimization, with the optimization objective being the minimization of Lyapunov drift penalty.
[0011] Using the aforementioned multimodal offloading scheduling system model, a modal offloading strategy is generated;
[0012] According to the modality offloading strategy, the edge device obtains a first modality recognition result based on the first spectral data, adjusts the resolution of the second spectral data, and then transmits it to the server. The server obtains a second modality recognition result based on the second spectral data.
[0013] At the edge device, the first modality recognition result and the second modality recognition result are probabilistically integrated to obtain a prediction result.
[0014] Secondly, an electronic device includes:
[0015] Memory is used to store executable instructions or computer programs.
[0016] A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method described in the first aspect.
[0017] Thirdly, a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the method as described in the first aspect.
[0018] Fourthly, a computer program product includes a computer program or computer-executable instructions, which, when executed by a processor, implement the method described in the first aspect.
[0019] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:
[0020] This invention establishes a multimodal scheduling framework comprising an edge device and a server, along with a Lyapunov-optimized multimodal offloading scheduling system model. It utilizes an early exit mechanism to obtain several early exit nodes for the first modality model, minimizing the Lyapunov drift penalty as the optimization objective. The multimodal offloading scheduling system model generates a modality offloading strategy. The edge device then generates a first modality recognition result based on the modality offloading strategy, adjusts the resolution of the second spectral data, and transmits it to the server for processing. At the edge device, the first and second modality recognition results are probabilistically integrated based on very late fusion to obtain a prediction result. Compared to existing technologies, this invention reduces the computational overhead at the edge device. By leveraging the multimodal characteristics of computationally limited edge devices, it ensures system stability and model processing accuracy while improving system processing speed. Attached Figure Description
[0021] Figure 1 This is a flowchart illustrating the multimodal scheduling method in Embodiment 1 of this application;
[0022] Figure 2 This is a schematic diagram of the multimodal scheduling framework in Embodiment 1 of this application;
[0023] Figure 3 This is a schematic diagram of a thermal model with an early exit node in Embodiment 1 of this application. Detailed Implementation
[0024] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses. The term "determine" broadly covers a wide variety of actions, including acquiring, calculating, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), probing, and similar actions; it may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and similar actions; it may also include generating, creating, establishing, and similar actions; and parsing, selecting, choosing, and similar actions, etc. Definitions of other terms will be given in the following description.
[0025] It should be noted that when one element is considered to be "connected" to another element, it can be directly connected to the other element or connected to the other element through an intermediary element. Furthermore, in the following embodiments, "connection" should be understood as "electrical connection," "communication connection," etc., if there is transmission of electrical signals or data between the connected objects.
[0026] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.
[0027] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;
[0028] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.
[0029] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0030] Example 1
[0031] In this embodiment, infrared modal images are used as the first spectral data, RGB modal images are used as the second spectral data, a thermal model is used as the first modal model, and an RGB model is used as the second modal model. This provides a multimodal scheduling method based on early exit for edge-constrained devices, applicable to the field of multispectral object recognition. (See reference...) Figure 1 ,include:
[0032] S10. Establish a multimodal scheduling framework, see [reference needed]. Figure 2 The system includes an edge device and a server; wherein the edge device is equipped with a first modal model for processing first spectral data to obtain a first modal recognition result, and the server is equipped with a second modal model for processing second spectral data to obtain a second modal recognition result.
[0033] S20. Obtain several early exit nodes for the first modal model using the early exit mechanism.
[0034] S30. Under the maximum delay constraint, based on the network environment information and the early exit node, a multimodal offloading scheduling system model is established based on Lyapunov optimization, with the optimization objective being the minimization of Lyapunov drift penalty.
[0035] S40. Using the multimodal offloading scheduling system model, generate a modal offloading strategy.
[0036] S50. According to the modal offloading strategy, the edge device obtains the first modal recognition result based on the first spectral data, adjusts the resolution of the second spectral data, and transmits it to the server. The server obtains the second modal recognition result based on the second spectral data.
[0037] S60. At the edge device, the first modality recognition result and the second modality recognition result are probabilistically integrated to obtain a prediction result.
[0038] Those skilled in the art will understand that in multimodal collaborative processing, feature fusion is mainly divided into early fusion, mid-term fusion, and very late fusion. Early fusion starts processing two modal data together early in the data processing stage; mid-term fusion processes the two modal data separately first, and then fuses the intermediate feature results in the middle; very late fusion uses a probabilistic embedding method to fuse the results of the two modal processing. In this embodiment, very late fusion is used as a multimodal scheduling scenario.
[0039] It should be noted that the first modal model is a network model based on the chain rule, meaning that the result of the calculation in the previous stage of the model will be passed to the next stage, and finally an output result is obtained, as shown in the following formula:
[0040] F(x)=f L (f L-1 (…(f1(x)))) (1)
[0041] It should also be noted that the scheduling method needs to dynamically schedule the inputs of each modality according to the environmental conditions, so as to ensure both model accuracy and improve system processing speed. This embodiment considers adjusting the RGB image resolution at the edge device and then offloading it to the server for processing. The impact of different image resolutions on model accuracy can be viewed as a convex function, as shown in the following equation:
[0042]
[0043] In the formula, A1 represents the result of the RGB model (i.e., the first modal model), a, b, and d are the fitting parameters, and e is the base of the natural logarithm.
[0044] In some specific implementation processes, the impact of different resolutions on the accuracy of the RGB model was obtained by testing the FLIR_ADAS_v2 dataset, and the impact of different resolutions was used as initial values for Lyapunov optimization.
[0045] It is understood that this embodiment can reduce the computational overhead of edge devices. By utilizing the multimodal characteristics of edge devices with limited computing resources, the input of multimodal models can be scheduled according to the current computing power of the device and the network status, which can not only ensure system stability and model processing accuracy, but also improve the system processing speed.
[0046] In some preferred embodiments, the first model is trained using a preset training set to obtain a number of early exit nodes; wherein the early exit nodes are represented by the model's average accuracy and are marked with the corresponding first model processing time.
[0047] It is understandable that by using an early exit mechanism to control the training of the first modality model, multiple early exit nodes can be obtained.
[0048] Specifically, the average accuracy r of the model at each exit point was obtained by testing on the FLIR_ADAS_v2 dataset. n and calculation time tt n (i.e., the first model processing time), thermal models with early exit nodes, such as Figure 3 As shown.
[0049] It should be noted that, due to network fluctuations, while considering the highest RGB resolution (image frame quality) for unloading can yield high-precision results, it also comes at the cost of significant latency. Therefore, Lyapunov optimization is used to ensure the stability of the system.
[0050] In some alternative embodiments, the Lyapunov optimization includes:
[0051] Based on Lyapunov optimization, a virtual queue Q(t) is introduced, with an initial value of 0. The expression for the state at time t+1 is as follows:
[0052] Q(t+1)=max{Q(t)+L all -L max ,0} (3)
[0053] L all =max(L rgb ,tt n (4)
[0054] L rgb =l tran +s (5)
[0055]
[0056] In the formula, L max L represents the maximum delay constraint. all Indicates the total system time spent; tt nIndicates selection r n The first model processing time required by the first modal model when it is an early exit node in the schedule at time t; L rgb The delay represents the time required for the second spectral data to be unloaded to the server and processed; s represents the server-side processing time for the second model; l tran R represents the transmission delay of the second spectral data being unloaded to the server; R represents the size of the second spectral data that needs to be transmitted; B represents the transmission delay. t This represents the network bandwidth at time t.
[0057] For each time slot t, define a quadratic Lyapunov function L(t):
[0058]
[0059] The Lyapunov drift penalty is introduced, and the minimum of the Lyapunov drift penalty is used as the optimization objective when generating the modal unloading strategy, so that the multimodal scheduling framework satisfies the model accuracy and latency in the long time T.
[0060] Furthermore, a precision surcharge is introduced to adjust the Lyapunov drift penalty. The adjusted Lyapunov drift penalty expression is as follows:
[0061] Δ(Q(t)) ′ =Δ(Q(t))-ωA t (8)
[0062] In the formula, Δ(Q(t)) ′ Δ(Q(t)) represents the adjusted Lyapunov drift penalty; Δ(Q(t)) represents the unadjusted Lyapunov drift penalty; ωA t This indicates the precision additional item, ω represents the adjustment parameter, and A t This indicates the recognition accuracy of the second spectral data.
[0063] It is understandable that in practical applications, the system needs to satisfy both model accuracy and latency over a long time T. In the above embodiment, to ensure model accuracy, a precision-based term is added to the Lyapunov algorithm, using ω as an adjustment parameter. Each time the system makes a decision, it needs to minimize the Lyapunov penalty. Obviously, when the latency is large, the penalty according to the formula is also large; when the latency is small, improving accuracy can bring better long-term results to the system. Through Lyapunov optimization, the system can guarantee certain constraints on accuracy and latency. When the system has high requirements for model accuracy, the weight of model accuracy can be increased by increasing the value of ω.
[0064] Furthermore, the generated modal unloading strategy includes:
[0065] Obtain the Lyapunov drift penalty at the corresponding time point, and determine whether the Lyapunov drift penalty is higher than a preset threshold value Q. max :
[0066] If so, when selecting the early exit point of the first modal model, a time not exceeding L is used. rgb The most recent early exit point;
[0067] Otherwise, based on the overall model accuracy and total system time of the multimodal scheduling framework, the optimal computational cost-effectiveness is determined, and based on the optimal cost-effectiveness, the early exit point and the resolution of the second spectral data unloaded to the server are determined as the modal unloading strategy.
[0068] Furthermore, the expression for the optimal computational cost-effectiveness is:
[0069]
[0070] In the formula, P(A1,A2) represents the overall model accuracy; A1 represents the model accuracy of the first modal model under the corresponding modal unloading strategy; and A2 represents the model accuracy of the second modal model under the corresponding modal unloading strategy.
[0071] In some specific implementations, the pseudocode for the multimodal adaptive offloading method on the edge device side is shown below:
[0072] Table 1. Pseudocode of the Adaptive Unloading Method
[0073]
[0074] Assuming x1 represents the output of the RGB model, x2 represents the output of the thermal model, y represents the object label, and p represents the final predicted probability (i.e., the prediction result), the effectiveness of the probability ensemble is shown below:
[0075] p(x1,x2|y)=p(x1|y)p(x2|y) (10)
[0076]
[0077] In some preferred embodiments, the process expression for the probability integration is as follows:
[0078]
[0079] In the formula, x i represents the information of the i-th modality; p(·) represents the prediction result; M represents the total number of modalities; y represents the object label.
[0080] It should be emphasized that this probability-based probabilistic ensemble method can be extended to more modalities, demonstrating its scalability.
[0081] In some preferred embodiments, to facilitate the continuous operation of the multimodal scheduling framework, the virtual queue Q at time slot t+1 is continuously updated. t+1 Repeat steps S30-S60 to achieve adaptive dynamic scheduling.
[0082] Example 2
[0083] This embodiment provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor, causing the processor to perform some or all of the steps of the method provided in Embodiment 1 of this application.
[0084] It is understood that the storage medium can be transient or non-transient. Exemplarily, the storage medium includes, but is not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0085] By way of example, the processor may be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.
[0086] By way of example, the read-only memory includes, but is not limited to, MASK ROM, PROM, EPROM, EEPROM, Flash, etc.
[0087] By way of example, the random access memory includes, but is not limited to, DRAM, SRAM, SDRAM, DDR SDRAM, etc.
[0088] In some examples, a computer program product is provided, which can be implemented by hardware, software, or a combination thereof. As a non-limiting example, the computer program product can be embodied in the storage medium, or it can be embodied in a software product, such as an SDK (Software Development Kit).
[0089] As a non-limiting example, a computer program product is provided, comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium, and executes the computer-executable instructions, causing the electronic device to perform some or all of the steps of the method described in the embodiments of this application.
[0090] In some examples, a computer program is provided, including computer-readable code, wherein, when the computer-readable code is run in a computer device, a processor in the computer device performs some or all of the steps for implementing the method.
[0091] This embodiment also proposes an electronic device, including a memory and a processor. The memory stores at least one instruction, at least one program, code set, or instruction set. When the processor executes the at least one instruction, at least one program, code set, or instruction set, it implements some or all of the steps of the method described in Embodiment 1.
[0092] In some examples, a hardware entity of the electronic device is provided, including: a processor, a memory, and a communication interface; wherein the processor typically controls the overall operation of the electronic device; the communication interface is used to enable the electronic device to communicate with other terminals or servers via a network; the memory is configured to store instructions and applications executable by the processor, and may also cache data to be processed or already processed (including but not limited to image data, audio data, voice communication data, and video communication data) to be processed by the processor and various modules in the electronic device, and may be implemented using flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or random access memory (RAM).
[0093] A processor may include one or more processing elements. Therefore, a processor may include one or more integrated circuits (ICs) configured to perform the functions of the processor. Furthermore, each integrated circuit may include circuitry (e.g., a first circuit, a second circuit, and other circuitry) configured to perform the functions of the processor.
[0094] Furthermore, data can be transferred between the processor, communication interface, and memory via a bus, which can include any number of interconnected buses and bridges, connecting various circuits of one or more processors and memories together.
[0095] It is understood that the options in Embodiment 1 above also apply to this embodiment, so they will not be described again here.
[0096] The same or similar labels correspond to the same or similar parts;
[0097] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this application.
[0098] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.
[0099] In different specific implementations, the methods or systems described in this application can be implemented in software, hardware, or a combination thereof. Furthermore, the order of the method steps can be changed, and various elements can be added, reordered, combined, omitted, or modified.
[0100] Obviously, the above embodiments of this application are merely examples for clearly illustrating this application, and are not intended to limit the implementation of this application, nor are they intended to limit this application. For those skilled in the art, other variations or modifications can be made based on the above description. The separate structural / functional modules or units can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. The structure and function of the separate components can be implemented as a combined structure or component. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of the claims of this application.
Claims
1. A multimodal scheduling method based on early exit for edge-constrained devices, characterized in that, Applications include: multispectral object recognition A multimodal scheduling framework is established, including an edge device and a server; wherein, the edge device is deployed with a first modal model and the server is deployed with a second modal model; An early exit mechanism is used to obtain several early exit nodes for the first modal model; wherein the first modal model is a network model based on the chain rule, and the first modal model is trained with a preset training set to obtain several early exit nodes; and the early exit nodes are represented by the model's average accuracy and are marked with the corresponding processing time of the first modal model. Under the maximum delay constraint, based on network environment information and the early exit node, a multimodal offloading scheduling system model is established based on Lyapunov optimization, with the optimization objective being the minimization of Lyapunov drift penalty. Using the aforementioned multimodal offloading scheduling system model, a modal offloading strategy is generated; According to the modality offloading strategy, the edge device uses the first modality model to obtain a first modality recognition result based on the first spectral data, adjusts the resolution of the second spectral data, and then transmits it to the server. The server uses the second modality model to obtain a second modality recognition result based on the second spectral data. At the edge device, the first modality recognition result and the second modality recognition result are probabilistically integrated to obtain a prediction result; The Lyapunov optimization includes: Based on Lyapunov optimization, a virtual queue Q(t) is introduced, with an initial value of 0. The expression for the state at time t+1 is as follows: Q(t+1)=max{Q(t)+L all -L max ,0} L all =max(L rgb ,tt n ) L rgb =l tran +s In the formula, L max L represents the maximum delay constraint. all Indicates the total system time spent; tt n Indicates selection r n The first model processing time required by the first modal model when it is an early exit node in the schedule at time t; L rgb The delay represents the time required for the second spectral data to be unloaded to the server and processed; s represents the server-side processing time for the second model; L tran R represents the transmission delay of the second spectral data being unloaded to the server; R represents the size of the second spectral data that needs to be transmitted; B represents the transmission delay. t This represents the network bandwidth at time t.
2. The multimodal scheduling method based on early exit for edge-constrained devices according to claim 1, characterized in that, For each time slot t, define a quadratic Lyapunov function L(t): The Lyapunov drift penalty is introduced, and the minimum of the Lyapunov drift penalty is used as the optimization objective when generating the modal unloading strategy, so that the multimodal scheduling framework satisfies the model accuracy and latency in the long time T.
3. The multimodal scheduling method based on early exit for edge-constrained devices according to claim 1, characterized in that, An accuracy surcharge is introduced to adjust the Lyapunov drift penalty. The adjusted Lyapunov drift penalty expression is as follows: Δ(Q(t)) ′ =Δ(Q(t))-ωA t In the formula, Δ(Q(t)) ′ Δ(Q(t)) represents the adjusted Lyapunov drift penalty; Δ(Q(t)) represents the unadjusted Lyapunov drift penalty; ωA t This indicates the precision additional item, ω represents the adjustment parameter, and A t This indicates the recognition accuracy of the second spectral data.
4. The multimodal scheduling method based on early exit for edge-constrained devices according to claim 3, characterized in that, The generated modal unloading strategy includes: Obtain the Lyapunov drift penalty at the corresponding time point, and determine whether the Lyapunov drift penalty is higher than a preset threshold value Q. max : If so, when selecting the early exit node of the first modal model, the time should not exceed L. rgb The most recent early exit node; Otherwise, based on the overall model accuracy and total system time of the multimodal scheduling framework, the optimal computational cost-effectiveness is determined, and the resolution of the early exit node and the second spectral data unloaded to the server is determined as the modal unloading strategy based on the optimal computational cost-effectiveness.
5. A multimodal scheduling method based on early exit for edge-constrained devices according to claim 4, characterized in that, The expression for the optimal cost-effectiveness calculation is: In the formula, P(A1,A2) represents the overall model accuracy; A1 represents the model accuracy of the first modal model under the corresponding modal unloading strategy; and A2 represents the model accuracy of the second modal model under the corresponding modal unloading strategy.
6. A multimodal scheduling method based on early exit for edge-constrained devices according to any one of claims 1-5, characterized in that, The process expression for the probability integration is as follows: In the formula, x i represents the information of the i-th modality; p(·) represents the prediction result; M represents the total number of modalities; y represents the object label.
7. An electronic device, characterized in that, include: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1-6.
8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by a processor to implement the method as described in any one of claims 1-6.
9. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the method described in any one of claims 1-6 is implemented.