A large model operation method, device, equipment and medium based on edge device

Abstract

The present application discloses a large model operation method, apparatus, equipment and medium based on edge devices, which relates to the field of artificial intelligence technology, including: preprocessing the large model to be run, and optimizing the preprocessed large model to be run based on the hardware characteristics of the edge device; determining the task priority of the tasks to be executed corresponding to the optimized model based on the current resource situation of the edge device; allocating the tasks to be executed to a pre-built model collaborative operation framework according to the task priority, and analyzing the tasks to be executed during the allocation process, so as to deploy the tasks to be executed to the load balancing node of the model collaborative operation framework according to the analysis results; managing relevant parameter data using a preset data cache management algorithm, so as to preload the data to be used in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding tasks to be executed based on the data to be used through the load balancing node of the model collaborative operation framework.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large model operation method, device, equipment and medium based on edge devices. Background Art

[0002] Currently, the application of large models on edge devices faces many challenges. On the one hand, existing edge computing solutions mostly use static task allocation, which leads to low task processing efficiency when resources are tight, and inability to fully utilize device capabilities when resources are sufficient. On the other hand, the caching strategy of model parameters and intermediate results in existing technologies is relatively simple. During the operation of large models, the need to temporarily load data increases the running time, reduces the running efficiency, and further affects the user experience. On the one hand, when processing complex tasks, the delay problem is more prominent. On the other hand, the development of the hardware capabilities of edge devices is limited, and their hardware capabilities cannot be fully utilized, especially in low-power, high-performance scenarios. In other words, existing technologies limit the widespread application of large models on edge devices, with unreasonable resource allocation, low operating efficiency, and limited application scenarios.

[0003] In summary, how to optimize the running process of large models on edge devices to solve application problems caused by the shortcomings of current technology is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a large model operation method, apparatus, device, and medium based on edge devices, which can optimize the operation process of large models on edge devices to solve the application problems caused by the shortcomings of current technologies. The specific solution is as follows:

[0005] In a first aspect, the present application provides a large model operation method based on edge devices, comprising:

[0006] Preprocessing the large model to be run, and optimizing the preprocessed large model to be run based on the hardware characteristics of the edge device to obtain an optimized model;

[0007] Determine the priority of the task to be executed corresponding to the optimized model based on the current resource status of the edge device;

[0008] Allocating the tasks to be executed to a pre-built model collaborative operation framework according to the task priority, and analyzing the tasks to be executed during the allocation process, so as to deploy the tasks to the load balancing nodes of the model collaborative operation framework based on the obtained analysis results; the model collaborative operation framework is a framework constructed by the edge device, edge server and cloud;

[0009] A preset data cache management algorithm is used to manage the relevant parameter data required to run the optimized model, so as to preload the data to be used in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding tasks to be executed based on the data to be used through the load balancing node of the model collaborative operation framework; the data to be used is the data to be used in the next step in the process of running the optimized model, which is predicted by the preset data cache management algorithm.

[0010] Optionally, the preprocessing of the large model to be run includes:

[0011] The weights of the large model to be run are compressed using a preset quantization technology, and redundant parameters in the large model to be run are removed using a preset pruning technology to obtain a pruned large model;

[0012] The knowledge of the pruned large model is transferred to the small model through a preset knowledge distillation technology, and the model structure of the transferred model is redesigned and the parameters are adjusted to obtain an adjusted model;

[0013] mining the sparse characteristics of the adjusted model using a gradient-based sparse training method to obtain a compressed model with a sparse structure;

[0014] Accordingly, the optimization of the pre-processed large model to be run based on the hardware characteristics of the edge device includes:

[0015] The compressed model forming a sparse structure is optimized based on the hardware characteristics of the edge device.

[0016] Optionally, the optimizing the pre-processed large model to be run based on the hardware characteristics of the edge device to obtain an optimized model includes:

[0017] Utilizing the characteristics of the field programmable gate array in the edge device, mapping the pre-processed target computing module in the large model to be run to the field programmable gate array;

[0018] Calculating a computing frequency of a computing engine corresponding to the target computing module based on a clock speed and a pipeline delay rate of the field programmable gate array, so as to design the computing engine according to the computing frequency;

[0019] If it is detected that a graphics processor is configured in the edge device, the single instruction multiple data architecture of the graphics processor is used to calculate the sub-batch size based on the total batch size of the batch data and the number of cores of the graphics processor, so as to split the batch data according to the sub-batch size, and distribute the split data to the cores of the graphics processor for parallel computing to obtain an optimized model.

[0020] Optionally, determining the task priority of the to-be-executed task corresponding to the optimized model based on the current resource status of the edge device includes:

[0021] Determine a corresponding task type weight according to the task type of the to-be-executed task corresponding to the optimized model, and determine a corresponding task priority requirement weight based on whether the to-be-executed task satisfies a user-defined task priority condition;

[0022] Determining a corresponding urgency weight according to the urgency of the task to be executed, and determining a corresponding resource dependency weight according to the dependency of the task to be executed on resources;

[0023] Calculating the task priority of the task to be executed based on the task type weight, the task priority processing requirement weight, the urgency weight, and the resource dependency weight;

[0024] Accordingly, allocating the tasks to be executed to the pre-built model collaborative operation framework according to the task priorities includes:

[0025] When the task priority is not lower than the first preset priority, the task to be executed is assigned to the edge device in the pre-built model collaborative operation framework;

[0026] When the task priority is not lower than the second preset priority and lower than the first preset priority, the task to be executed is assigned to the edge server in the model collaborative operation framework;

[0027] When the task priority is lower than the second preset priority, the task to be executed is allocated to the cloud in the model collaborative operation framework;

[0028] The first preset priority is higher than the second preset priority.

[0029] Optionally, before allocating the tasks to be executed to the pre-built model collaborative operation framework according to the task priorities, the method further includes:

[0030] When the edge device and the edge server in the model collaborative operation framework are started, their respective resource information is registered with the cloud in the model collaborative operation framework through the edge device and the edge server respectively, so that the cloud manages the distribution of the tasks to be executed according to the resource information and the task priority; the edge device, the edge server and the cloud in the model collaborative operation framework communicate with each other through a preset communication protocol.

[0031] Optionally, analyzing the tasks to be executed to deploy the tasks to be executed to a load balancing node of the model collaborative operation framework according to the obtained analysis results includes:

[0032] Analyze the tasks to be executed to determine the computational complexity, data volume, and real-time requirements of the tasks to be executed, and determine the relevant performance indicators corresponding to the load balancing nodes of the model collaborative operation framework;

[0033] Based on the computational complexity, the data volume, the real-time requirements and the relevant performance indicators, the task allocation fitness of the task to be executed on the corresponding load balancing node is calculated respectively, so as to allocate the task to be executed to the target load balancing node according to the task allocation fitness; the target load balancing node is the load balancing node corresponding to the target task allocation fitness among the several load balancing nodes, and the target task allocation fitness is the task allocation fitness with the smallest value among the task allocation fitnesses of the task to be executed corresponding to each load balancing node.

[0034] Optionally, managing the relevant parameter data required to run the optimized model using a preset data cache management algorithm to preload the to-be-used data in the relevant parameter data into the edge device includes:

[0035] Calculating the data access frequency of the relevant parameter data required to run the optimized model using an exponentially weighted moving average method, and determining the importance of the relevant parameter data;

[0036] Cache the relevant parameter data into a target cache location according to the importance and the data access frequency; the target cache location includes a memory cache location, a flash memory cache location, and a cloud cache location;

[0037] Extracting and analyzing features of the cached data in the target cache location to predict a predicted access frequency of the cached data in the next step using a preset autoregressive moving average model;

[0038] Determine a corresponding cache location score based on the relevant information of the cache data stored in the target cache location;

[0039] The cache value of the cache data is calculated based on the cache location score, the predicted access frequency and the importance of the cache data, so as to adjust the cache location of the cache data according to the cache value. At the same time, the user behavior prediction data output by the preset user behavior model and the current status of the model collaborative operation framework are used to determine the data to be used in the cache data, so as to preload the data to be used into the memory cache location of the edge device.

[0040] Optionally, the process of executing the corresponding task to be executed based on the data to be used further includes:

[0041] When the data to be used needs to be transmitted in the model collaborative operation framework, a pre-processing operation of compression and encapsulation is performed on the data to be used to obtain processed data;

[0042] Determining the data priority and data type of the processed data, and calculating a target channel bandwidth for transmitting the processed data based on the task priority of the task to be executed and the total available bandwidth;

[0043] determining a target transmission channel for the processed data according to the data priority, the data type, and the target channel bandwidth, so as to transmit the processed data using the target transmission channel;

[0044] Furthermore, during the process of transmitting the processed data, a check value of the processed data is calculated using a preset data check algorithm, so as to determine whether an abnormality occurs in the transmission of the processed data according to the check value.

[0045] Optionally, the process of executing the corresponding task to be executed based on the data to be used further includes:

[0046] Continuously monitoring the task queue of the edge server;

[0047] When it is detected that there is an urgent task in the task queue, pausing or adjusting other tasks in the task queue except the urgent task;

[0048] Determine the corresponding resource allocation ratio according to the target task priority of the urgent task, and determine the target computing resource amount of the urgent task and the current available computing resource amount of the edge server;

[0049] Calculating a resource allocation amount for the urgent task based on the resource allocation ratio, the target computing resource amount, and the currently available computing resource amount, so as to allocate corresponding computing resources from the other tasks according to the obtained resource allocation amount, so as to utilize the computing resources to execute the urgent task;

[0050] If, after allocating the computing resources, it is detected that the current edge server is unable to perform the emergency task through the computing resources, the target computing link of the emergency task is offloaded to the cloud, so that the cloud can assist the edge server in performing the emergency task.

[0051] In a second aspect, the present application provides a large model operation device based on an edge device, comprising:

[0052] A model optimization module is used to preprocess the large model to be run and optimize the preprocessed large model to be run based on the hardware characteristics of the edge device to obtain an optimized model;

[0053] A task priority determination module, configured to determine the task priority of the task to be executed corresponding to the optimized model based on the current resource status of the edge device;

[0054] A task analysis module is used to allocate the pending tasks to a pre-built model collaborative operation framework according to the task priority, and analyze the pending tasks during the allocation process to deploy the pending tasks to the load balancing nodes of the model collaborative operation framework based on the obtained analysis results; the model collaborative operation framework is a framework constructed by the edge device, edge server and cloud;

[0055] A data management module is used to manage the relevant parameter data required to run the optimized model using a preset data cache management algorithm, so as to preload the data to be used in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding tasks to be executed based on the data to be used through the load balancing node of the model collaborative operation framework; the data to be used is the data to be used in the next step in the process of running the optimized model, which is predicted by the preset data cache management algorithm.

[0056] In a third aspect, the present application provides an electronic device, comprising:

[0057] Memory, used to store computer programs;

[0058] A processor is used to execute the computer program to implement the aforementioned large model operation method based on edge devices.

[0059] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned large model operation method based on edge devices is implemented.

[0060] In this embodiment, the large model to be run is preprocessed, and the preprocessed large model to be run is optimized based on the hardware characteristics of the edge device to obtain an optimized model; based on the current resource situation of the edge device, the task priority of the task to be executed corresponding to the optimized model is determined; the task to be executed is assigned to a pre-built model collaborative operation framework according to the task priority, and during the assignment process, the task to be executed is analyzed to deploy the task to the load balancing node of the model collaborative operation framework according to the obtained analysis results; the model collaborative operation framework is a framework constructed by the edge device, edge server and cloud; the preset data cache management algorithm is used to manage the relevant parameter data required to run the optimized model, so as to preload the data to be used in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding task to be executed based on the data to be used through the load balancing node of the model collaborative operation framework; the data to be used is the data to be used in the next step in the process of running the optimized model predicted by the preset data cache management algorithm. As can be seen from the above, the present application first pre-processes the large model to be run, and optimizes the pre-processed large model to be run according to the hardware characteristics of the edge device, and at the same time determines the task priority of the tasks to be executed of the optimized model according to the current resource situation of the edge device, so as to allocate the tasks to be executed to the model collaborative operation framework according to the task priority, and in the allocation process, deploy them to the load balancing node of the model collaborative operation framework according to the analysis results of the tasks to be executed, and at the same time use the preset data cache management algorithm to manage the relevant parameter data, so as to pre-load the data to be used in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding tasks to be executed based on the data to be used through the load balancing node of the model collaborative operation framework. In this way, through the above process of this application, the large model to be run is optimized based on the hardware characteristics of the edge device, which can give full play to the hardware capabilities of the edge device to improve the efficiency of the large model operation; the task priority of the task to be executed is determined according to the resource situation of the edge device, and the task to be executed is dynamically allocated according to the task priority, which can fully and reasonably realize resource allocation and improve system performance; the preset data cache management algorithm is used to manage the relevant parameter data, and the data to be used is preloaded, and the relevant data is loaded into the cache of the edge device in advance, which reduces the waiting time of model inference and can improve the operation efficiency; building a model collaborative operation framework consisting of edge devices, edge servers and cloud, can effectively solve the delay problem, and then optimize the operation process of the large model on the edge device to solve the application problems caused by the shortcomings of the current technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0062] Figure 1 This is a flow chart of a large model operation method based on edge devices disclosed in this application;

[0063] Figure 2 This is a flow chart of a specific large model operation method based on edge devices disclosed in this application;

[0064] Figure 3 This is a schematic diagram of the structure of a large model operation device based on edge devices disclosed in this application;

[0065] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0067] Currently, the application of large models on edge devices faces many challenges. On the one hand, existing edge computing solutions mostly use static task allocation, which leads to low task processing efficiency when resources are tight, and inability to fully utilize device capabilities when resources are sufficient. On the other hand, the caching strategy of model parameters and intermediate results in existing technologies is relatively simple. During the operation of large models, the need to temporarily load data increases the running time, reduces the running efficiency, and further affects the user experience. On the one hand, when processing complex tasks, the delay problem is more prominent. On the other hand, the development of the hardware capabilities of edge devices is limited, and their hardware capabilities cannot be fully utilized, especially in low-power, high-performance scenarios. In other words, existing technologies limit the widespread application of large models on edge devices, with unreasonable resource allocation, low operating efficiency, and limited application scenarios.

[0068] In order to overcome the above technical problems, the present application provides a large model operation method based on edge devices to optimize the operation process of large models on edge devices to solve application problems caused by the shortcomings of current technology.

[0069] See also Figure 1 As shown, an embodiment of the present invention discloses a large model operation method based on an edge device, including:

[0070] Step S11: pre-process the large model to be run, and optimize the pre-processed large model to be run based on the hardware characteristics of the edge device to obtain an optimized model.

[0071] In this embodiment, when a large model needs to be run on an edge device, in order to reduce the number of parameters and computational complexity of the large model, the large model to be run is first preprocessed to make it more suitable for running on the resource-constrained edge device. The preprocessed large model to be run is then optimized based on the hardware characteristics of the edge device to obtain an optimized model, further improving the model's inference speed. The large models to be run include, but are not limited to, large models in the fields of manufacturing, quality control, and process optimization; the hardware characteristics include, but are not limited to, graphics processing units (GPUs) and field programmable gate arrays (FPGAs).

[0072] It should be pointed out that the preprocessing operations include but are not limited to quantization, pruning, knowledge distillation, etc., and the processing flow is as follows: the weights of the large model to be run are compressed using a preset quantization technology, and redundant parameters in the large model to be run are removed through a preset pruning technology to obtain a pruned large model; the knowledge of the pruned large model is transferred to the small model through a preset knowledge distillation technology, and the model structure of the migrated model is redesigned and the parameters are adjusted to obtain an adjusted model; the sparse characteristics of the adjusted model are mined using a gradient-based sparse training method to obtain a compressed model with a sparse structure; accordingly, the preprocessed large model to be run is optimized based on the hardware characteristics of the edge device, including: optimizing the compressed model with a sparse structure based on the hardware characteristics of the edge device. That is, the preset quantization technology is used to compress the weights of the large model to be run, so as to compress its weights from 32-bit floating point numbers to 8-bit or lower precision integers. At the same time, the preset pruning technology is used to remove redundant parameters in the large model to be run to reduce the model volume and obtain a pruned large model. Subsequently, the preset knowledge distillation technology is used to migrate the knowledge of the pruned large model to a lightweight small model to improve the performance of the small model. The migrated model is structurally reparameterized, and its model structure is redesigned and parameters are adjusted to make it more suitable for running on edge devices while maintaining high accuracy to obtain an adjusted model. Specifically, the convolution layer of the convolutional neural network of the migrated model can be reparameterized, and multiple convolution kernels can be fused into an equivalent convolution kernel to reduce the number of parameters. The fusion formula can be specifically:

[0073] ;

[0074] in, is the weight coefficient; is the i-th convolution kernel; Represents the equivalent convolution kernel after fusion. After obtaining the adjusted model, the adjusted model is subjected to sparsity utilization processing, and a gradient-based sparse training method is used to exploit the sparse characteristics in the adjusted model. Through sparsity calculation and storage optimization, the computational complexity and storage requirements of the model are further reduced. The sparsity of the weights is dynamically adjusted during the training process to obtain a compressed model with a naturally sparse structure. The sparsity calculation formula can be specifically:

[0075] ;

[0076] Where s represents sparsity; non_zero_params is a non-zero parameter used to indicate the number of non-zero weight parameters in the model; and total_params is a total parameter used to indicate the total number of weight parameters in the model. Therefore, in the step of optimizing the pre-processed large model to be run based on the hardware characteristics of the edge device, the pre-processed large model to be run is the compressed model with a sparse structure.

[0077] It should be further pointed out that, after pre-processing the large model to be run, this embodiment can customize and optimize the pre-processed large model to be run according to the hardware characteristics of the edge device to give full play to the computing power of the hardware. The processing flow is as follows: utilizing the characteristics of the field programmable gate array in the edge device, the target computing module in the pre-processed large model to be run is mapped to the field programmable gate array; based on the clock speed and pipeline delay rate of the field programmable gate array, the computing frequency of the computing engine corresponding to the target computing module is calculated to design the computing engine according to the computing frequency; if it is detected that a graphics processor is configured in the edge device, the single instruction multiple data architecture of the graphics processor is used to calculate the sub-batch size based on the total batch size of the batch data and the number of cores of the graphics processor, so as to split the batch data according to the sub-batch size, and distribute the split data to the cores of the graphics processor for parallel computing to obtain the optimized model. Wherein, the target computing module includes but is not limited to convolutional layers, fully connected layers, etc. That is, since edge devices are usually equipped with FPGAs, the programmable logic and hardware acceleration capabilities of the FPGA can be used to map the pre-processed target computing module in the large model to be run to the FPGA to achieve efficient hardware acceleration. Specifically, the computing frequency of the computing engine corresponding to the target computing module can be calculated based on the clock speed and pipeline delay rate of the FPGA, so as to design the computing engine according to the computing frequency. For example, the convolution layer in the model can be mapped to the programmable logic unit of the FPGA to design a dedicated convolution computing engine. The calculation formula for the computing frequency can be specifically:

[0078] ;

[0079] Wherein, f represents the actual calculation frequency of the convolution calculation engine of the FPGA; Clock_Speed ​​is the clock speed of the FPGA; Pipeline_Delay_Rate is the pipeline delay rate, that is, the delay rate in the FPGA pipeline design. This embodiment can optimize the pipeline design through the calculated calculation frequency, reduce delay, and improve calculation throughput. In addition, if it is detected that a GPU is configured in the edge device, this embodiment can make full use of the GPU's SIMD (single instruction multiple data) architecture to transform the model for parallel computing optimization to accelerate computationally intensive tasks such as matrix operations. Specifically, the batch data can be divided into multiple sub-batches and assigned to different GPU cores for parallel computing. The sub-batch size of the sub-batch can be calculated based on the total batch size of the batch data and the number of GPU cores, so as to divide the batch data according to the sub-batch size and distribute the divided data to the cores of the graphics processor for parallel computing to obtain the optimized model. The calculation formula of the sub-batch size can be specifically:

[0080] ;

[0081] Wherein, b is the sub-batch size, which is used to indicate the sub-batch size allocated to each GPU core after the total batch of batch data is split; Total_Batch_Size is the total batch size, which is used to indicate the total batch size during model training or inference; GPU_Core_Count is the number of GPU cores, which is used to indicate the number of cores available in the GPU. In this way, when a large model needs to be run on an edge device, this embodiment pre-processes the large model to be run through technologies such as quantization, pruning, knowledge distillation, structural re-parameterization, and sparsity utilization, which can reduce the number of parameters, computational complexity, and storage requirements of the large model while maintaining high accuracy, making it more suitable for running on resource-constrained edge devices; based on the hardware characteristics of the edge device, the pre-processed large model to be run is customized and optimized to give full play to the computing power of the hardware, achieve efficient hardware acceleration, reduce latency, improve computing throughput, and further improve the inference speed of the model.

[0082] Step S12: Based on the current resource situation of the edge device, determine the task priority of the task to be executed corresponding to the optimized model.

[0083] In this embodiment, after obtaining the optimized model, the task priority of the to-be-executed task corresponding to the optimized model can be determined based on the current resource status of the edge device, so that the to-be-executed task can be assigned according to the obtained task priority, and the task can be intelligently determined to be more suitable for execution locally, in the cloud, or on the edge server, thereby achieving efficient collaborative processing of tasks. The resource status refers to the real-time resource status of the edge device, including but not limited to CPU (Central Processing Unit) usage, memory occupancy, network bandwidth, etc.

[0084] It should be pointed out that this embodiment can comprehensively consider a variety of factors to determine the task priority, and its processing flow is as follows: determine the corresponding task type weight based on the task type of the task to be executed corresponding to the optimized model, and determine the corresponding task priority requirement weight based on whether the task to be executed meets the user-defined task priority processing condition; determine the corresponding urgency weight based on the urgency of the task to be executed, and determine the corresponding resource dependence weight based on the degree of dependence of the task to be executed on resources; calculate the task priority of the task to be executed based on the task type weight, the task priority requirement weight, the urgency weight, and the resource dependence weight. Among them, the user-defined task priority processing condition refers to whether the task to be executed is a task that the user explicitly requires to be prioritized. That is, the corresponding task type weight is determined according to the task type of the task to be executed corresponding to the optimized model. For example, tasks with high real-time requirements such as equipment status monitoring and fault warning have relatively high priority, and their task type weight can be set to 0.6; batch processing tasks such as data statistical analysis have relatively low task priority, and their task type weight can be set to 0.3; the corresponding task priority processing requirement weight is determined based on whether the task to be executed meets the user-defined task priority processing conditions. If the user explicitly requires priority processing of the task, its priority will be increased accordingly, and its task priority processing requirement weight can be set to 0.5, and the task priority processing requirement weight of ordinary tasks can be set to 0.2; the corresponding urgency is determined by the urgency of the task to be executed Weight, for example, the priority of emergency tasks for sudden fault handling is higher than that of routine tasks, the urgency weight of emergency tasks can be set to 0.7, and the urgency weight of routine tasks can be set to 0.3; the corresponding resource dependency weight is determined according to the degree of dependence of the task to be executed on resources. For tasks with high resource requirements, if the current resources are tight, their priority will be dynamically adjusted, where the resource dependency weight of high resource dependency tasks can be set to 0.4, and the resource dependency weight of low resource dependency tasks can be set to 0.1. After determining the task type weight, the task priority processing requirement weight, the urgency weight, and the resource dependency weight, the task priority of the task to be executed is calculated using the above weights. The calculation formula for the task priority can be specifically:

[0085] ;

[0086] in, is the task type weight; Prioritize demand weights for said tasks; is the urgency weight; The weight of resource dependency is used. This embodiment determines task priorities based on the current resource availability of edge devices and relevant information about pending tasks, taking into account multiple factors including task type, task priority requirements, task urgency, and task resource dependency. This optimizes resource allocation, meets business needs, ensures the timely execution of critical tasks, and improves the operational efficiency of large models.

[0087] Step S13: Allocate the tasks to be executed to a pre-built model collaborative operation framework according to the task priority, and during the allocation process, analyze the tasks to be executed so as to deploy the tasks to the load balancing node of the model collaborative operation framework according to the analysis results; the model collaborative operation framework is a framework constructed by the edge device, edge server and cloud.

[0088] In this embodiment, the corresponding tasks to be executed are assigned to a pre-built model collaborative operation framework based on the determined task priority, and when assigned, the tasks to be executed are analyzed so as to deploy the tasks to be executed to the load balancing node of the model collaborative operation framework according to the obtained analysis results. Among them, the model collaborative operation framework is a unified end-edge-cloud collaborative framework constructed by the edge devices, edge servers and cloud, which can support seamless data transmission and task collaboration between edge devices, edge servers and cloud; the cloud serves as the collaborative control center in the model collaborative operation framework, which is used to reasonably allocate the tasks to be executed to each load balancing node according to task requirements and resource conditions, and coordinate the task execution order and data interaction between each load balancing node to ensure efficient completion of tasks.

[0089] It is understood that before assigning pending tasks to the pre-built model collaborative operation framework, the framework must first be established. Specifically, a unified communication protocol must be established to standardize the data transmission format, communication process, and error handling mechanism between edge devices, edge servers, and the cloud, ensuring seamless communication between all nodes. For example, the packet format can be defined as follows: Header (20 bytes): Contains data type, source address, destination address, timestamp, etc. Payload (message body) (variable length): Actual transmitted data. Footer (4 bytes): Contains a checksum. Resource discovery and registration are then performed, and the process flow is as follows: When the edge device and edge server in the model collaborative operation framework are started, they register their respective resource information with the cloud in the model collaborative operation framework through the edge device and edge server, respectively, so that the cloud can manage the allocation of pending tasks based on the resource information and task priority. The edge device, edge server, and cloud in the model collaborative operation framework communicate with each other using a pre-set communication protocol. The resource information includes, but is not limited to, device type, computing power, storage capacity, etc. That is, when the edge device and the edge server in the model collaborative operation framework are started, they respectively register their own resource information with the cloud in the model collaborative operation framework, so that the cloud maintains a global resource directory based on the resource information and the task priority, and manages the allocation of the tasks to be executed, including query and allocation during task scheduling.

[0090] It should be pointed out that after determining the task priority, the processing flow of allocating the task to be executed to the pre-built model collaborative operation framework according to the task priority is as follows: when the task priority is not lower than the first preset priority, the task to be executed is allocated to the edge device in the pre-built model collaborative operation framework; when the task priority is not lower than the second preset priority and is lower than the first preset priority, the task to be executed is allocated to the edge server in the model collaborative operation framework; when the task priority is lower than the second preset priority, the task to be executed is allocated to the cloud in the model collaborative operation framework; wherein, the first preset priority is higher than the second preset priority. That is, the tasks to be executed are allocated according to the set first preset priority and the second preset priority. For example, the first preset priority can be set to 0.6, and the second preset priority can be set to 0.3. When the task priority, that is, the P value, is not lower than 0.6, the tasks to be executed are allocated to the edge device in the pre-built model collaborative operation framework, and are preferentially executed locally on the edge device. If the local resources are insufficient, they are allocated to the neighboring edge server. If the local resources of the neighboring edge server are still insufficient, they are allocated to the dedicated cloud resources; when the P value is not lower than 0.3 and lower than 0.6, the tasks to be executed are allocated to the edge server in the model collaborative operation framework, and are preferentially executed on the edge server. If the edge server resources are tight, part of the tasks to be executed are allocated to the cloud; when the P value is lower than 0.3, the tasks to be executed are allocated to the cloud in the model collaborative operation framework, and are mainly processed in batches on the cloud. It can be understood that this embodiment can continuously collect the resource usage of the edge device, determine the task priority based on the obtained resource monitoring data, and dynamically allocate tasks based on the task priority. For example, for tasks with tight resources, some tasks can be automatically offloaded to the cloud or edge server; for devices with sufficient resources, more tasks are performed locally as much as possible. By dynamically adjusting the distribution of tasks between edge devices and the cloud, resource utilization can be optimized and latency can be reduced.

[0091] It should be further pointed out that in the process of allocating the tasks to be executed, it is also necessary to perform task allocation of the load balancing nodes of the model collaborative operation framework based on the analysis of the tasks to be executed. The processing flow is as follows: Analyze the tasks to be executed to determine the computational complexity, data volume and real-time requirements of the tasks to be executed, and determine the relevant performance indicators corresponding to the several load balancing nodes of the model collaborative operation framework; calculate the task allocation fitness of the tasks to be executed at the corresponding load balancing nodes based on the computational complexity, data volume, real-time requirements and relevant performance indicators, so as to allocate the tasks to be executed to the target load balancing node according to the task allocation fitness; the target load balancing node is the load balancing node corresponding to the target task allocation fitness among the several load balancing nodes, and the target task allocation fitness is the task allocation fitness with the smallest value among the task allocation fitness of the tasks to be executed corresponding to each load balancing node. That is, the tasks to be executed are analyzed to determine the computational complexity, data volume, and real-time requirements of the tasks to be executed, and the resource status of the edge devices, edge servers, and cloud are monitored in real time, including but not limited to CPU, memory, network bandwidth, etc., so as to evaluate the processing capacity of each load balancing node based on the above dynamic data, and determine the relevant performance indicators corresponding to the several load balancing nodes of the model collaborative operation framework, so as to calculate the task allocation fitness of the tasks to be executed on the corresponding load balancing nodes according to the above factors. That is, for each of the tasks to be executed, its task allocation fitness on all available nodes is calculated. Since the smaller the value of the task allocation fitness, the higher the fitness of the task to be executed on the node, that is, the more suitable the node is for executing the task, the task to be executed can be assigned to the load balancing node corresponding to the target task allocation fitness among the several load balancing nodes, wherein the target task allocation fitness is the task allocation fitness with the smallest value among the task allocation fitness of the tasks to be executed corresponding to each load balancing node. The calculation formula of the task allocation fitness can be specifically:

[0092] ;

[0093] Wherein, C represents the computational complexity; D represents the data volume; T represents the real-time requirement; Node_Capability is the processing capability, i.e., the computing capability of the load balancing node, which can be expressed as the number of operations that can be executed per second or the benchmark score; Node_Storage is the storage capability, i.e., the storage capacity of the load balancing node, in bytes; Node_Latency is the latency characteristic, i.e., the average latency of the load balancing node processing tasks, which can be expressed as a normalized value (between 0 and 1), where 0 represents the lowest latency. In this way, after determining the task priority, this embodiment dynamically allocates tasks to be executed according to the task priority, which can optimize resource utilization and reduce latency; a model collaborative operation framework constructed by edge devices, edge servers, and the cloud is established, which can achieve efficient collaboration of data between edge devices, edge servers, and the cloud, ensure seamless communication between nodes, and thereby more efficiently execute tasks to be executed and improve processing efficiency; tasks are reasonably allocated to edge devices, edge servers, and the cloud based on resource status and task characteristics, avoiding single-point overload and resource waste.

[0094] Step S14: Use a preset data cache management algorithm to manage the relevant parameter data required to run the optimized model, so as to preload the data to be used in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding tasks to be executed based on the data to be used through the load balancing node of the model collaborative operation framework; the data to be used is the data to be used in the next step in the process of running the optimized model, which is predicted by the preset data cache management algorithm.

[0095] In this embodiment, a preset data cache management algorithm is used to manage the relevant parameter data required to run the optimized model, so that the data to be used in the relevant parameter data is preloaded into the cache of the edge device in advance, reducing data transmission delays and improving task processing efficiency, so that the optimized model can execute the corresponding tasks to be executed based on the data to be used through the load balancing node of the model collaborative operation framework. The relevant parameter data includes the model parameters and intermediate results of the optimized model; the data to be used is the data that will be used in the next step of running the optimized model, as predicted by the preset data cache management algorithm.

[0096] It should be pointed out that the memory cache can store data with high access frequency and high importance, such as key parameters of the currently executing task, frequently interacted data, etc., to ensure fast access; the flash cache can store data with medium access frequency and high importance, such as recently used model parameters, more important intermediate results, etc., to ensure fast reading and writing speeds and long-term preservation; the cloud cache can store data with low access frequency and relatively low importance, such as historical data, backup data, etc., so that it can be obtained on demand through the network, saving edge device storage resources. Therefore, the processing flow of managing relevant parameter data using the preset data cache management algorithm is as follows: the data access frequency of the relevant parameter data required to run the optimized model is calculated using the exponentially weighted moving average method, and the importance of the relevant parameter data is determined; the relevant parameter data is cached to the target cache location according to the importance and the data access frequency; the target cache location includes the memory cache location, the flash cache location and the cloud cache location. That is, the data access frequency of the relevant parameter data is calculated using the exponentially weighted moving average method, and the importance of the relevant parameter data is determined. Among them, the calculation formula for the data access frequency can be specifically:

[0097] ;

[0098] in, is the access frequency at time t. If there is no historical data, its value is set to 0.5. If there is historical data, the actual data at t-1 is used. α is the smoothing coefficient, which determines the weight distribution between the historical AF value and the current number of visits. The default value is 0.2. After running for a period of time, the absolute value of the slope of the access frequency change curve in the first 30 minutes is taken as the smoothing coefficient. is the number of data accesses at time t. After determining the data access frequency and the importance, the relevant parameter data is cached in a memory cache location, a flash cache location, or a cloud cache location based on the importance and the data access frequency. For example, when the data access frequency of the current data is not less than 0.7 and the importance is high, the current data may be stored in the memory cache location; when the data access frequency of the current data is less than 0.7 but not less than 0.3 and the importance is high or medium, the current data may be stored in the flash cache location; when the data access frequency of the current data is less than 0.3 and the importance is low, the current data may be stored in the cloud cache location.

[0099] It should be further pointed out that after managing the relevant parameter data, this embodiment can analyze user behavior and environmental perception information through a machine learning algorithm to predict the user's possible next operation, and preload the data to be used in the relevant parameter data into the cache of the edge device in advance. The processing flow is as follows: feature extraction and analysis are performed on the cache data in the target cache location to predict the next predicted access frequency of the cache data through a preset autoregressive moving average model; the corresponding cache location score is determined based on the relevant information of the cache data stored in the target cache location; the cache value of the cache data is calculated based on the cache location score, the predicted access frequency and the importance of the cache data, so as to adjust the cache location of the cache data according to the cache value, and at the same time, the user behavior prediction data output by the preset user behavior model and the current status of the model collaborative operation framework are used to determine the data to be used in the cache data, so as to preload the data to be used into the memory cache location of the edge device. Wherein, the features include but are not limited to the access history, relevance, timeliness, etc. of the cached data; the preset autoregressive moving average model is used to predict the future access frequency of the data; the relevant information includes storage cost and access speed; the preset user behavior model is used to predict the data that the user may need at a specific point in time, which can be a model based on time series analysis, including a Markov chain model, and the transition probability matrix in the Markov chain model is obtained through historical data statistics. That is, feature extraction and analysis are performed on the cached data in the target cache location to extract the access time series of the data, and a preset autoregressive moving average model (i.e., ARIMA model, Autoregressive Integrated Moving Average Model) is established so as to predict the next predicted access frequency of the cached data through the preset autoregressive moving average model. Wherein, the formula of the preset autoregressive moving average model can be specifically:

[0100] ;

[0101] Among them, B is the lag operator; is obtained by performing i consecutive operations on B, indicating that the time series is lagged by order i; φ is the autoregressive coefficient, which indicates the impact of the past p-period observations on the current value; θ is the moving average coefficient, which indicates the impact of the past q-period forecast errors on the current value; d is the difference order; is a white noise error. A corresponding cache location score is then determined based on the relevant information of the cache data stored in the target cache location, and the cache value of the cache data is calculated based on the cache location score, the predicted access frequency, and the importance of the cache data. The calculation formula for the cache value can be specifically:

[0102] ;

[0103] Wherein, Predicted_AF is the predicted access frequency, that is, the future data access frequency predicted by the ARIMA model; V represents the cache value; Cache_Location_Score is the cache location score, that is, the score calculated based on the storage cost and access speed of the data in different cache locations. After obtaining the cache value, the cache location and retention strategy of the data can be determined based on the cache value. It can be understood that the higher the cache value, the higher the access frequency, importance and cache location score of the data. Therefore, for data with the highest cache value, it can be stored in the memory cache location first to achieve fast access; data with medium cache value has relatively low access frequency and importance, but still requires a faster access speed and is suitable for storage in the flash cache location; data with the lowest cache value has low access frequency and low importance and can be stored in the cloud cache location to save storage resources of the edge device. At the same time, this embodiment can use the user behavior prediction data output by the preset user behavior model and the current state of the model collaborative operation framework to determine the data to be used in the cache data, so as to preload the data to be used into the memory cache location of the edge device. The user behavior prediction data is obtained by outputting the preset user behavior model. This embodiment can collect user behavior data, including user-system interaction records such as clickstream data, query history, and operation time. This user behavior data can be used to construct the preset user behavior model to analyze user usage habits and preferences. For example, in an industrial scenario, this embodiment can record the frequency and patterns of device parameter adjustments made by equipment operators over different time periods. This collected user behavior data can then be used to train a machine learning model to predict the user's likely next action. For example, by analyzing the operator's device parameter adjustment patterns over the past week, this embodiment can predict which parameters the operator may need to adjust during the next workday and pre-load these parameters into the cache. It will be appreciated that this embodiment can continuously monitor data access and dynamically adjust cache management policies to ensure the cache is always optimal. For example, data predicted to be accessed soon can be pre-loaded into the memory cache; data that has not been accessed for a long time and is of low importance can be migrated from the memory cache to the flash cache or cloud cache.

[0104] It should be noted that this embodiment works collaboratively through the model collaborative operation framework, and the edge server in the model collaborative operation framework can assist in processing some urgent tasks to ensure the operation of the optimized large model. Therefore, the processing flow in the process of executing the corresponding tasks to be executed based on the data to be used can also be as follows: continuously monitoring the task queue of the edge server; when it is detected that there is an urgent task in the task queue, pausing or adjusting other tasks in the task queue except the urgent task; determining the corresponding resource allocation ratio according to the target task priority of the urgent task, and determining the target computing resource amount of the urgent task and the current available computing resource amount of the edge server; calculating the resource allocation amount of the urgent task based on the resource allocation ratio, the target computing resource amount and the current available computing resource amount, so as to allocate corresponding computing resources from the other tasks according to the obtained resource allocation amount, so as to use the computing resources to execute the urgent task; if after allocating the computing resources, it is detected that the current edge server cannot execute the urgent task through the computing resources, then the target computing link of the urgent task is offloaded to the cloud, so as to use the cloud to assist the edge server in executing the urgent task. Among them, the target calculation link is a key calculation link. That is, the edge server adopts the task queue method to manage tasks, wherein the task processing order follows the principle of high to low priority, and the task queue of the edge server is monitored in real time. When it is detected that there is an urgent task such as equipment failure warning processing in the task queue, other tasks in the task queue except the urgent task are suspended or adjusted, and the urgent task is inserted into the head of the task queue. At the same time, the corresponding resource allocation ratio is determined according to the target task priority of the urgent task, and the target computing resource amount of the urgent task and the current available computing resource amount of the edge server are determined, so as to calculate the resource allocation amount of the urgent task based on the resource allocation ratio, the target computing resource amount and the current available computing resource amount. The calculation formula of the resource allocation amount can be specifically:

[0105] ;

[0106] Among them, Emergency_Task_Requirement represents the target computing resource amount, that is, the computing resource amount required for the emergency task; Current_Resource is the currently available computing resource amount, that is, the computing resource amount that the edge server can currently use to process the task; Resource_Allocation_Ratio represents the resource allocation ratio, that is, the ratio of resources allocated from other tasks. After obtaining the resource allocation amount, this embodiment can temporarily allocate corresponding computing resources from the other tasks according to the resource allocation amount to ensure that the emergency task has sufficient computing resources, and then use the computing resources to execute the emergency task. In addition, if after allocating the computing resources, if it is detected that the current edge server cannot execute the emergency task through the computing resources, that is, the resources are still insufficient, it can quickly communicate with the cloud to offload the target computing link of the emergency task to the cloud, so as to use the powerful computing power of the cloud to complete the calculation, and quickly feed back the obtained calculation results to the edge device, thereby assisting the edge server to execute the emergency task. In this way, this embodiment uses the results of the preset user behavior model output to load the data and model parameters that may be needed from the cloud or low-speed storage to the high-speed cache of the edge device in advance, which can significantly reduce data access delay and improve task processing efficiency and response speed; cache data in a hierarchical manner according to the access frequency and importance of the data, thereby optimizing the user experience; continuously monitor the data access situation and the status of the model collaborative operation framework to dynamically adjust the cache management strategy to ensure that the cache is always in the optimal state; assist in processing some urgent tasks through the edge server in the model collaborative operation framework to ensure the operation of the optimized large model and improve operation efficiency.

[0107] As can be seen from the above, the embodiment of the present application first pre-processes the large model to be run, and optimizes the pre-processed large model to be run according to the hardware characteristics of the edge device, and at the same time determines the task priority of the tasks to be executed of the optimized model according to the current resource situation of the edge device, so as to allocate the tasks to be executed to the model collaborative operation framework according to the task priority, and in the allocation process, deploy them to the load balancing node of the model collaborative operation framework according to the analysis results of the tasks to be executed, and at the same time use the preset data cache management algorithm to manage the relevant parameter data, so as to pre-load the data to be used in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding tasks to be executed based on the data to be used through the load balancing node of the model collaborative operation framework.In this way, through the above process of the embodiment of the present application, on the one hand, when a large model needs to be run on an edge device, the large model to be run is preprocessed through technologies such as quantization, pruning, knowledge distillation, structural reparameterization, and sparsity utilization, which can reduce the number of parameters, computational complexity, and storage requirements of the large model while maintaining high accuracy, making it more suitable for running on resource-constrained edge devices; on the other hand, the preprocessed large model to be run is customized and optimized based on the hardware characteristics of the edge device to give full play to the computing power of the hardware, achieve efficient hardware acceleration, reduce latency, increase computing throughput, and further improve the reasoning speed of the model; on the other hand, based on the current resource situation of the edge device and the relevant information of the task to be executed, the task priority is determined by comprehensively considering multiple factors such as task type, task priority processing requirements, task urgency, and task resource dependence, which can optimize resource allocation, meet business needs, ensure the timely execution of key tasks, and improve the operating efficiency of the large model; on the one hand, after determining the task priority, the tasks to be executed are dynamically allocated according to the task priority, which can optimize resource utilization and reduce latency; on the other hand, a large model to be run is built by edge devices and edge servers. The model collaborative operation framework built on the edge device, edge server and cloud can achieve efficient collaboration of data between edge devices, edge servers and the cloud, ensure seamless communication between nodes, and thus more efficiently execute tasks to be executed and improve processing efficiency; on the one hand, tasks are reasonably allocated to edge devices, edge servers and the cloud according to resource status and task characteristics to avoid single point overload and resource waste; on the other hand, the results of the preset user behavior model output are used to load the required data and model parameters from the cloud or low-speed storage to the high-speed cache of the edge device in advance, which can significantly reduce data access delay and improve task processing efficiency and response speed; on the other hand, data is cached in a hierarchical manner according to the access frequency and importance of the data, thereby optimizing the user experience; on the one hand, the data access situation and the status of the model collaborative operation framework are continuously monitored to dynamically adjust the cache management strategy to ensure that the cache is always in the optimal state; on the other hand, the edge server in the model collaborative operation framework assists in processing some urgent tasks to ensure the operation of the optimized large model, improve operation efficiency, and thus optimize the operation process of the large model on the edge device to solve the application problems caused by the shortcomings of current technology.

[0108] Based on the above embodiments, it can be seen that through the method of the present application, in the process of executing the corresponding pending tasks based on the predicted data to be used in the next step of the optimization process of the large model, the data can be transmitted based on the preset data transmission mechanism when it needs to be transmitted in the model collaborative operation framework to improve the efficiency of data transmission. For this reason, this embodiment describes in detail how to perform data transmission based on the preset data transmission mechanism. Figure 2 As shown, an embodiment of the present invention discloses a large model operation method based on an edge device, including:

[0109] Step S21: When the data to be used needs to be transmitted in the model collaborative operation framework, a pre-processing operation of compression and encapsulation is performed on the data to be used to obtain processed data.

[0110] In this embodiment, when data to be used needs to be transmitted within the model collaborative operation framework, it can be compressed to reduce the amount of data to be transmitted and then pre-processed for encapsulation to facilitate transmission and parsing, thereby obtaining processed data. The data to be used is the data predicted by a preset data cache management algorithm to be used in the next step of running the large optimization model; the model collaborative operation framework is a framework constructed by edge devices, edge servers, and the cloud. Specifically, when compressing the data to be used, this embodiment can utilize a modified LZ77 algorithm (a dictionary-based lossless compression algorithm). This algorithm improves compression efficiency by adding a hash table and pre-processing steps. Frequently occurring strings in the data to be used are counted to create a dictionary. During compression, matching strings can be quickly found using the dictionary and replaced with dictionary indexes, achieving efficient compression of the data to be used and reducing the amount of data to be transmitted. Furthermore, during the pre-processing of encapsulating the data to be used, the data to be used can be encapsulated into a data packet in a specific format. The data packet contains information such as data content, source address, destination address, and priority, facilitating transmission and parsing. In this way, when the data to be used needs to be transmitted in the model collaborative operation framework, this embodiment performs pre-processing operations of compression and encapsulation on it, so as to reduce the amount of transmitted data through compression to more efficiently utilize the bandwidth, reduce the transmission time, and improve the efficiency of data transmission and data analysis through encapsulation.

[0111] Step S22: Determine the data priority and data type of the processed data, and calculate the target channel bandwidth for transmitting the processed data based on the task priority of the task to be executed and the total available bandwidth.

[0112] In this embodiment, the data priority and data type of the processed data are determined, and the target channel bandwidth for transmitting the processed data is calculated based on the task priority of the task to be executed corresponding to the processed data and the total available bandwidth. The calculation formula for the target channel bandwidth can be specifically:

[0113] ;

[0114] in, The bandwidth allocated to the i-th channel, is the task priority of the task to be executed, is the sum of all task priorities, is the total available bandwidth. It can be understood that in this embodiment, multiple data transmission channels are established for data transmission, and the target channel bandwidth is calculated so that a suitable channel can be selected for transmission of the data to be processed according to the target channel bandwidth. In this way, this embodiment calculates the target channel bandwidth for transmitting the processed data based on the task priority of the task to be executed and the total available bandwidth, thereby realizing the flexible determination of the target channel bandwidth according to the characteristics of the task, which is convenient for adapting to the complex and changing edge computing environment.

[0115] Step S23: determining a target transmission channel for the processed data according to the data priority, the data type, and the target channel bandwidth, so as to transmit the processed data using the target transmission channel.

[0116] In this embodiment, the target transmission channel of the processed data is determined based on the data priority, the data type, and the target channel bandwidth, so that the processed data can be transmitted using the target transmission channel. For example, for data with high real-time requirements, a high-speed, low-latency data transmission channel can be selected as its target transmission channel, so that the data can be transmitted using the high-speed, low-latency data transmission channel; for batch data, a data transmission channel with a larger bandwidth but a slightly higher latency can be selected as its target transmission channel, so that the data can be transmitted using the data transmission channel with a larger bandwidth but a slightly higher latency, thereby dynamically allocating transmission channels based on task priority and data type. In this way, this embodiment determines the target transmission channel of the processed data based on the data priority, data type, and target channel bandwidth, which can more efficiently utilize bandwidth and improve data transmission efficiency.

[0117] Step S24: During the transmission of the processed data, a check value of the processed data is calculated using a preset data check algorithm, so as to determine whether an abnormality occurs in the transmission of the processed data according to the check value.

[0118] In this embodiment, during the process of transmitting the processed data using the target transmission channel, a checksum value of the processed data can be calculated using a preset data checksum algorithm, so as to determine whether an abnormality has occurred in the transmission of the processed data based on the checksum value, thereby facilitating timely retransmission upon discovery of transmission errors and ensuring data integrity. The preset data checksum algorithm can be a CRC-32 checksum algorithm.

[0119] Specifically, this embodiment can calculate a checksum value for the processed data during transmission and verify it at the receiving end based on the checksum value. If the checksum fails, it means that an abnormality has occurred in the transmission of the processed data. A retransmission strategy can be determined based on the importance and urgency of the processed data. Regarding importance, if the abnormal processed data is extremely important, its loss could result in significant losses such as system crashes. Therefore, a retransmission operation can be triggered immediately. To ensure successful retransmission of such data, a backup channel can be established for parallel transmission. If the abnormal processed data is less important, it means that the data has little impact on the system and can be processed during idle time or directly discarded. Regarding urgency, if the abnormal processed data has extremely high real-time requirements, it means that it needs to arrive within a strict time limit. Therefore, a retransmission timeout can be dynamically adjusted to quickly attempt multiple retransmissions. If the abnormal processed data has low real-time requirements, a certain delay can be allowed, or retries can be made during idle time. It should be noted that this embodiment can also use a data synchronization mechanism during the transmission of the processed data to ensure data consistency. In this way, this embodiment utilizes the data synchronization and verification mechanism to effectively reduce the data transmission error rate and improve the reliability of data transmission.

[0120] As can be seen from the above, in an embodiment of the present application, when the data to be used needs to be transmitted in a model collaborative operation framework, the data to be used is first pre-processed by compression and encapsulation to obtain processed data, and then the data priority and data type of the processed data are determined. At the same time, the target channel bandwidth for transmitting the processed data is calculated based on the task priority of the task to be executed and the total available bandwidth, so as to determine the target transmission channel for the processed data according to the data priority, the data type and the target channel bandwidth, and then use the target transmission channel to transmit the processed data. In addition, in the process of transmitting the processed data, a preset data verification algorithm is used to calculate the verification value of the processed data, so as to determine whether the transmission of the processed data is abnormal based on the verification value. In this way, through the above process of the embodiment of the present application, on the one hand, when the data to be used needs to be transmitted in the model collaborative operation framework, a pre-processing operation of compression and encapsulation is performed on it, so as to reduce the amount of transmitted data through compression, utilize bandwidth more efficiently, reduce transmission time, and improve the efficiency of data transmission and data parsing through encapsulation; on the other hand, the target channel bandwidth for transmitting the processed data is calculated based on the task priority of the task to be executed and the total available bandwidth, so as to realize flexible determination of the target channel bandwidth according to the characteristics of the task, and facilitate adaptation to the complex and changeable edge computing environment; on the one hand, the target transmission channel of the processed data is determined according to the data priority, data type and target channel bandwidth, which can more efficiently utilize bandwidth and improve the efficiency of data transmission; on the other hand, the use of data synchronization and verification mechanism can effectively reduce the data transmission error rate, improve the reliability of data transmission, and thus improve the efficiency of data transmission.

[0121] Based on the previous examples, this application discloses a large model operation method based on edge devices, which can optimize the operation process of large models on edge devices to solve application problems caused by the shortcomings of current technologies. Next, we will describe in detail the large model operation method based on edge devices in the smart factory application scenario.

[0122] Currently, edge devices in this embodiment (such as industrial controllers) need to run a complex, large model to monitor equipment status and product quality on the production line in real time. This embodiment first uses quantization technology to compress the weights of this large model, also known as the equipment status monitoring model, from 32-bit floating-point numbers to 8-bit integers, reducing the model size by approximately 75%. Pruning technology also further removes redundant parameters, reducing the number of model parameters by 30%. Knowledge distillation technology transfers knowledge from the large model to a lightweight, smaller model, ensuring that the inference accuracy of the smaller model reaches over 95% of that of the large model. Structural reparameterization and sparsity exploitation are then used to further compress the model. Finally, the model is customized and optimized for the industrial controller's built-in FPGA, further improving inference speed. These optimizations enable the industrial controller to efficiently run the equipment status monitoring model in low-resource environments, reducing inference time by 20% and significantly improving production efficiency. It is then detected that the CPU and memory resources of the industrial controller are tight, but the network bandwidth is sufficient. Based on this information, this embodiment can offload complex reasoning tasks in the model (such as feature extraction and classification in deep learning) to the cloud, while retaining simple preprocessing tasks (such as data collection and preliminary screening) for local execution, while ensuring seamless switching of tasks between the industrial controller and the cloud, thereby optimizing resource utilization and reducing latency without reducing user experience.

[0123] During daily operation, the industrial controller in this embodiment uses a preloading mechanism to analyze the production line's operating mode. It discovers that specific model parameters are required for equipment status monitoring when the production line starts running at 8:00 AM each day. Based on this prediction, the model collaborative operation framework automatically loads the relevant model parameters from the cloud or flash cache into the memory cache at approximately 7:50 AM each day. When the production line starts at 8:00 AM, the model parameters are already loaded, eliminating the need to wait for data to be loaded from the cloud. This significantly reduces the wait time for model inference and improves production efficiency. Furthermore, multiple industrial controllers on the production line collaborate through the model collaborative operation framework. When resources on an industrial controller are limited, this embodiment can offload some tasks to a nearby edge server or further offload them to the cloud. Furthermore, a data synchronization mechanism ensures data consistency between edge devices and the cloud. For example, if a device on the production line fails, the edge device can quickly upload the fault data to the cloud for in-depth analysis. Meanwhile, the edge server can assist in handling some urgent tasks to ensure the normal operation of the production line. This model collaborative operation framework enables smart factories to more efficiently handle complex tasks, improving production efficiency and equipment reliability.

[0124] Accordingly, see Figure 3 As shown, the embodiment of the present application also provides a large model operation device based on an edge device, including:

[0125] The model optimization module 11 is used to preprocess the large model to be run and optimize the preprocessed large model to be run based on the hardware characteristics of the edge device to obtain an optimized model;

[0126] A task priority determination module 12 is used to determine the task priority of the to-be-executed task corresponding to the optimized model based on the current resource status of the edge device;

[0127] The task analysis module 13 is used to allocate the tasks to be executed to a pre-built model collaborative operation framework according to the task priority, and analyze the tasks to be executed during the allocation process to deploy the tasks to the load balancing node of the model collaborative operation framework based on the obtained analysis results; the model collaborative operation framework is a framework built by the edge device, edge server and cloud;

[0128] The data management module 14 is used to manage the relevant parameter data required for running the optimized model by using a preset data cache management algorithm, so as to preload the data to be used in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding task to be executed based on the data to be used through the load balancing node of the model collaborative operation framework; the data to be used is the data to be used in the next step in the process of running the optimized model, which is predicted by the preset data cache management algorithm.

[0129] As can be seen from the above, the embodiment of the present application first pre-processes the large model to be run, and optimizes the pre-processed large model to be run according to the hardware characteristics of the edge device, and at the same time determines the task priority of the tasks to be executed of the optimized model according to the current resource situation of the edge device, so as to allocate the tasks to be executed to the model collaborative operation framework according to the task priority, and in the allocation process, deploy them to the load balancing node of the model collaborative operation framework according to the analysis results of the tasks to be executed, and at the same time use the preset data cache management algorithm to manage the relevant parameter data, so as to pre-load the data to be used in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding tasks to be executed based on the data to be used through the load balancing node of the model collaborative operation framework. In this way, through the above process of the embodiment of the present application, the large model to be run is optimized based on the hardware characteristics of the edge device, which can give full play to the hardware capabilities of the edge device to improve the efficiency of the large model operation; the task priority of the task to be executed is determined according to the resource situation of the edge device, and the task to be executed is dynamically allocated according to the task priority, which can fully and reasonably realize resource allocation and improve system performance; the preset data cache management algorithm is used to manage the relevant parameter data, and the data to be used is preloaded, and the relevant data is loaded into the cache of the edge device in advance, which reduces the waiting time of model inference and can improve the operation efficiency; constructing a model collaborative operation framework composed of edge devices, edge servers and cloud, can effectively solve the delay problem, and then optimize the operation process of the large model on the edge device to solve the application problems caused by the shortcomings of the current technology.

[0130] In some specific implementations, the model optimization module 11 may specifically include:

[0131] a parameter removal unit, configured to compress the weights of the large model to be run using a preset quantization technique, and remove redundant parameters in the large model to be run using a preset pruning technique to obtain a pruned large model;

[0132] A parameter adjustment unit, configured to transfer the knowledge of the pruned large model to the small model through a preset knowledge distillation technique, and to redesign the model structure and adjust the parameters of the transferred model to obtain an adjusted model;

[0133] A feature mining unit, configured to mine the sparse features of the adjusted model using a gradient-based sparse training method to obtain a compressed model with a sparse structure;

[0134] Accordingly, the model optimization module 11 may specifically include:

[0135] A model optimization unit is used to optimize the compressed model forming a sparse structure based on the hardware characteristics of the edge device.

[0136] In some specific implementations, the model optimization module 11 may specifically include:

[0137] A module mapping unit, configured to map the pre-processed target computing module in the large model to be run to the field programmable gate array by utilizing the characteristics of the field programmable gate array in the edge device;

[0138] a frequency calculation unit, configured to calculate a computing frequency of a computing engine corresponding to the target computing module based on a clock speed and a pipeline delay rate of the field programmable gate array, so as to design the computing engine according to the computing frequency;

[0139] A sub-batch size calculation unit is configured to calculate a sub-batch size based on the total batch size of the batch data and the number of cores of the graphics processor by utilizing the single instruction multiple data architecture of the graphics processor if it is detected that a graphics processor is configured in the edge device, so as to split the batch data according to the sub-batch size and distribute the split data to the cores of the graphics processor for parallel computing to obtain an optimized model.

[0140] In some specific implementations, the task priority determination module 12 may specifically include:

[0141] a processing requirement weight determination unit, configured to determine a corresponding task type weight according to the task type of the to-be-executed task corresponding to the optimized model, and determine a corresponding task priority processing requirement weight based on whether the to-be-executed task satisfies a user-defined task priority processing condition;

[0142] A dependency weight determination unit, configured to determine a corresponding urgency weight according to the urgency of the task to be executed, and determine a corresponding resource dependency weight according to the dependency of the task to be executed on the resource;

[0143] a task priority calculation unit, configured to calculate the task priority of the to-be-executed task based on the task type weight, the task priority processing requirement weight, the urgency weight, and the resource dependency weight;

[0144] Accordingly, the task analysis module 13 may specifically include:

[0145] A first task allocation unit is configured to allocate the task to be executed to the edge device in the pre-built model collaborative operation framework when the task priority is not lower than a first preset priority;

[0146] a second task allocating unit, configured to allocate the task to be executed to the edge server in the model collaborative operation framework when the task priority is not lower than a second preset priority and lower than the first preset priority;

[0147] a third task allocating unit, configured to allocate the to-be-executed task to the cloud in the model collaborative operation framework when the task priority is lower than the second preset priority;

[0148] The first preset priority is higher than the second preset priority.

[0149] In some specific implementations, the large model running apparatus based on edge devices may further include:

[0150] An information registration unit is used to register respective resource information to the cloud in the model collaborative operation framework through the edge device and the edge server respectively after the edge device and the edge server in the model collaborative operation framework are started, so that the cloud manages the allocation of the tasks to be executed according to the resource information and the task priority; the edge device, the edge server and the cloud in the model collaborative operation framework communicate with each other through a preset communication protocol.

[0151] In some specific implementations, the task analysis module 13 may specifically include:

[0152] An indicator determination unit is used to analyze the task to be executed to determine the computational complexity, data volume and real-time requirements of the task to be executed, and to determine the relevant performance indicators corresponding to the multiple load balancing nodes of the model collaborative operation framework;

[0153] A fitness calculation unit is used to calculate the task allocation fitness of the task to be executed on the corresponding load balancing node based on the computational complexity, the data volume, the real-time requirements and the relevant performance indicators, so as to allocate the task to be executed to the target load balancing node according to the task allocation fitness; the target load balancing node is the load balancing node corresponding to the target task allocation fitness among the several load balancing nodes, and the target task allocation fitness is the task allocation fitness with the smallest value among the task allocation fitness of the task to be executed corresponding to each load balancing node.

[0154] In some specific implementations, the data management module 14 may specifically include:

[0155] an importance determination unit, configured to calculate the data access frequency of the relevant parameter data required to run the optimized model using an exponentially weighted moving average method, and determine the importance of the relevant parameter data;

[0156] A data cache unit, configured to cache the relevant parameter data into a target cache location according to the importance and the data access frequency; the target cache location includes a memory cache location, a flash memory cache location, and a cloud cache location;

[0157] a data analysis unit, configured to extract and analyze features of the cached data in the target cache location, so as to predict a predicted access frequency of the cached data in the next step by using a preset autoregressive moving average model;

[0158] A score determining unit, configured to determine a corresponding cache location score based on relevant information of the cache data stored in the target cache location;

[0159] A value calculation unit is used to calculate the cache value of the cache data based on the cache location score, the predicted access frequency and the importance of the cache data, so as to adjust the cache location of the cache data according to the cache value, and at the same time use the user behavior prediction data output by the preset user behavior model and the current state of the model collaborative operation framework to determine the data to be used in the cache data, so as to preload the data to be used into the memory cache location of the edge device.

[0160] In some specific implementations, the large model running apparatus based on edge devices may further include:

[0161] A data preprocessing unit, configured to perform a preprocessing operation of compressing and encapsulating the data to be used to obtain processed data when the data to be used needs to be transmitted in the model collaborative operation framework;

[0162] a bandwidth calculation unit, configured to determine a data priority and a data type of the processed data, and calculate a target channel bandwidth for transmitting the processed data based on the task priority of the task to be executed and the total available bandwidth;

[0163] a channel determining unit, configured to determine a target transmission channel for the processed data according to the data priority, the data type, and the target channel bandwidth, so as to transmit the processed data using the target transmission channel;

[0164] Furthermore, during the process of transmitting the processed data, a check value of the processed data is calculated using a preset data check algorithm, so as to determine whether an abnormality occurs in the transmission of the processed data according to the check value.

[0165] In some specific implementations, the large model running apparatus based on edge devices may further include:

[0166] A queue monitoring unit, configured to continuously monitor the task queue of the edge server;

[0167] a task adjustment unit, configured to suspend or adjust other tasks in the task queue except the urgent task when detecting that there is an urgent task in the task queue;

[0168] A resource amount determination unit is used to determine a corresponding resource allocation ratio according to the target task priority of the urgent task, and to determine the target computing resource amount of the urgent task and the currently available computing resource amount of the edge server;

[0169] an allocation amount calculation unit, configured to calculate the resource allocation amount of the urgent task based on the resource allocation ratio, the target computing resource amount, and the currently available computing resource amount, so as to allocate corresponding computing resources from the other tasks according to the obtained resource allocation amount, so as to utilize the computing resources to execute the urgent task;

[0170] A link unloading unit is used to offload the target computing link of the emergency task to the cloud if it is detected that the current edge server cannot perform the emergency task through the computing resources after allocating the computing resources, so as to use the cloud to assist the edge server in performing the emergency task.

[0171] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the large model operation method based on edge devices disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0172] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0173] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0174] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to implement the large model operation method based on the edge device and executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to perform other specific tasks.

[0175] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned disclosed method for operating a large model based on an edge device. The specific steps of this method can be referred to the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0176] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0177] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0179] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0180] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A large model operation method based on edge devices, characterized in that: include: Preprocessing the large model to be run, and optimizing the preprocessed large model to be run based on the hardware characteristics of the edge device to obtain an optimized model; Calculate the task priority of the task to be executed corresponding to the optimized model based on the task type weight, the task priority processing requirement weight, the urgency weight, and the resource dependency weight; Allocating the tasks to be executed to a pre-built model collaborative operation framework according to the task priority, and analyzing the tasks to be executed during the allocation process, so as to deploy the tasks to the load balancing nodes of the model collaborative operation framework based on the obtained analysis results; the model collaborative operation framework is a framework constructed by the edge device, edge server and cloud; Managing the relevant parameter data required to run the optimized model using a preset data cache management algorithm to preload the to-be-used data in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding to-be-executed task based on the to-be-used data through the load balancing node of the model collaborative operation framework; The data to be used is the data to be used in the next step during the operation of the optimized model, as predicted by the preset data cache management algorithm; The method of managing the relevant parameter data required for running the optimized model by using a preset data cache management algorithm to preload the to-be-used data in the relevant parameter data into the edge device includes: Calculating the data access frequency of the relevant parameter data required to run the optimized model using an exponentially weighted moving average method, and determining the importance of the relevant parameter data; Cache the relevant parameter data into a target cache location according to the importance and the data access frequency; the target cache location includes a memory cache location, a flash memory cache location, and a cloud cache location; Extracting and analyzing features of the cached data in the target cache location to predict a predicted access frequency of the cached data in the next step using a preset autoregressive moving average model; Determine a corresponding cache location score based on the relevant information of the cache data stored in the target cache location; The cache value of the cache data is calculated based on the cache location score, the predicted access frequency and the importance of the cache data, so as to adjust the cache location of the cache data according to the cache value. At the same time, the user behavior prediction data output by the preset user behavior model and the current status of the model collaborative operation framework are used to determine the data to be used in the cache data, so as to preload the data to be used into the memory cache location of the edge device.

2. The large model operation method based on edge devices according to claim 1 is characterized in that: The preprocessing of the large model to be run includes: The weights of the large model to be run are compressed using a preset quantization technology, and redundant parameters in the large model to be run are removed using a preset pruning technology to obtain a pruned large model; The knowledge of the pruned large model is transferred to the small model through a preset knowledge distillation technology, and the model structure of the transferred model is redesigned and the parameters are adjusted to obtain an adjusted model; mining the sparse characteristics of the adjusted model using a gradient-based sparse training method to obtain a compressed model with a sparse structure; Accordingly, the optimization of the pre-processed large model to be run based on the hardware characteristics of the edge device includes: The compressed model forming a sparse structure is optimized based on the hardware characteristics of the edge device.

3. The large model operation method based on edge devices according to claim 1 is characterized in that: The optimizing the pre-processed large model to be run based on the hardware characteristics of the edge device to obtain an optimized model includes: Utilizing the characteristics of the field programmable gate array in the edge device, mapping the pre-processed target computing module in the large model to be run to the field programmable gate array; Calculating a computing frequency of a computing engine corresponding to the target computing module based on a clock speed and a pipeline delay rate of the field programmable gate array, so as to design the computing engine according to the computing frequency; If it is detected that a graphics processor is configured in the edge device, the single instruction multiple data architecture of the graphics processor is used to calculate the sub-batch size based on the total batch size of the batch data and the number of cores of the graphics processor, so as to split the batch data according to the sub-batch size, and distribute the split data to the cores of the graphics processor for parallel computing to obtain an optimized model.

4. The large model operation method based on edge devices according to claim 1 is characterized in that: Before calculating the task priority of the to-be-executed task corresponding to the optimized model based on the task type weight, the task priority processing requirement weight, the urgency weight, and the resource dependency weight, the method further includes: Determine a corresponding task type weight according to the task type of the to-be-executed task corresponding to the optimized model, and determine a corresponding task priority requirement weight based on whether the to-be-executed task satisfies a user-defined task priority condition; Determining a corresponding urgency weight according to the urgency of the task to be executed, and determining a corresponding resource dependency weight according to the dependency of the task to be executed on resources; Accordingly, allocating the tasks to be executed to the pre-built model collaborative operation framework according to the task priorities includes: When the task priority is not lower than the first preset priority, the task to be executed is assigned to the edge device in the pre-built model collaborative operation framework; When the task priority is not lower than the second preset priority and lower than the first preset priority, the task to be executed is assigned to the edge server in the model collaborative operation framework; When the task priority is lower than the second preset priority, the task to be executed is allocated to the cloud in the model collaborative operation framework; The first preset priority is higher than the second preset priority.

5. The large model operation method based on edge devices according to claim 1 is characterized in that: Before allocating the tasks to be executed to the pre-built model collaborative operation framework according to the task priorities, the method further includes: When the edge device and the edge server in the model collaborative operation framework are started, their respective resource information is registered with the cloud in the model collaborative operation framework through the edge device and the edge server respectively, so that the cloud manages the distribution of the tasks to be executed according to the resource information and the task priority; the edge device, the edge server and the cloud in the model collaborative operation framework communicate with each other through a preset communication protocol.

6. The large model operation method based on edge devices according to claim 1 is characterized in that: The analyzing the tasks to be executed and deploying the tasks to be executed to the load balancing nodes of the model collaborative operation framework according to the obtained analysis results includes: Analyze the tasks to be executed to determine the computational complexity, data volume, and real-time requirements of the tasks to be executed, and determine the relevant performance indicators corresponding to the load balancing nodes of the model collaborative operation framework; Based on the computational complexity, the data volume, the real-time requirements and the relevant performance indicators, the task allocation fitness of the task to be executed on the corresponding load balancing node is calculated respectively, so as to allocate the task to be executed to the target load balancing node according to the task allocation fitness; the target load balancing node is the load balancing node corresponding to the target task allocation fitness among the several load balancing nodes, and the target task allocation fitness is the task allocation fitness with the smallest value among the task allocation fitnesses of the task to be executed corresponding to each load balancing node.

7. The large model operation method based on edge devices according to claim 1 is characterized in that: The process of executing the corresponding task to be executed based on the data to be used further includes: When the data to be used needs to be transmitted in the model collaborative operation framework, a pre-processing operation of compression and encapsulation is performed on the data to be used to obtain processed data; Determining the data priority and data type of the processed data, and calculating a target channel bandwidth for transmitting the processed data based on the task priority of the task to be executed and the total available bandwidth; determining a target transmission channel for the processed data according to the data priority, the data type, and the target channel bandwidth, so as to transmit the processed data using the target transmission channel; Furthermore, during the process of transmitting the processed data, a check value of the processed data is calculated using a preset data check algorithm, so as to determine whether an abnormality occurs in the transmission of the processed data according to the check value.

8. The large model operation method based on edge devices according to claim 1 is characterized in that: The process of executing the corresponding task to be executed based on the data to be used further includes: Continuously monitoring the task queue of the edge server; When it is detected that there is an urgent task in the task queue, pausing or adjusting other tasks in the task queue except the urgent task; Determine the corresponding resource allocation ratio according to the target task priority of the urgent task, and determine the target computing resource amount of the urgent task and the current available computing resource amount of the edge server; Calculating a resource allocation amount for the urgent task based on the resource allocation ratio, the target computing resource amount, and the currently available computing resource amount, so as to allocate corresponding computing resources from the other tasks according to the obtained resource allocation amount, so as to utilize the computing resources to execute the urgent task; If, after allocating the computing resources, it is detected that the current edge server is unable to perform the emergency task through the computing resources, the target computing link of the emergency task is offloaded to the cloud, so that the cloud can assist the edge server in performing the emergency task.

9. A large model operation device based on edge devices, characterized in that: include: A model optimization module is used to preprocess the large model to be run and optimize the preprocessed large model to be run based on the hardware characteristics of the edge device to obtain an optimized model; A task priority calculation module is used to calculate the task priority of the to-be-executed task corresponding to the optimized model based on the task type weight, the task priority processing requirement weight, the urgency weight, and the resource dependency weight; A task analysis module is used to allocate the pending tasks to a pre-built model collaborative operation framework according to the task priority, and analyze the pending tasks during the allocation process to deploy the pending tasks to the load balancing nodes of the model collaborative operation framework based on the obtained analysis results; the model collaborative operation framework is a framework constructed by the edge device, edge server and cloud; A data management module, configured to manage the relevant parameter data required to run the optimized model using a preset data cache management algorithm, so as to preload the to-be-used data in the relevant parameter data into the edge device, so that the optimized model can execute the corresponding to-be-executed task based on the to-be-used data through the load balancing node of the model collaborative operation framework; The data to be used is the data to be used in the next step during the operation of the optimized model, as predicted by the preset data cache management algorithm; Wherein, the data management module includes: an importance determination unit, configured to calculate the data access frequency of the relevant parameter data required to run the optimized model using an exponentially weighted moving average method, and determine the importance of the relevant parameter data; A data cache unit, configured to cache the relevant parameter data into a target cache location according to the importance and the data access frequency; the target cache location includes a memory cache location, a flash memory cache location, and a cloud cache location; a data analysis unit, configured to extract and analyze features of the cached data in the target cache location, so as to predict a predicted access frequency of the cached data in the next step by using a preset autoregressive moving average model; A score determining unit, configured to determine a corresponding cache location score based on relevant information of the cache data stored in the target cache location; A value calculation unit is used to calculate the cache value of the cache data based on the cache location score, the predicted access frequency and the importance of the cache data, so as to adjust the cache location of the cache data according to the cache value, and at the same time use the user behavior prediction data output by the preset user behavior model and the current state of the model collaborative operation framework to determine the data to be used in the cache data, so as to preload the data to be used into the memory cache location of the edge device.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the large model operation method based on edge devices as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that Used to store computer programs; wherein, when the computer program is executed by a processor, the large model operation method based on edge devices as described in any one of claims 1 to 8 is implemented.

Images