Edge gateway computing power sharing method based on MQTT protocol

Abstract

The invention provides an edge gateway computing power sharing method based on an MQTT protocol. An edge terminal sends a computing power request to an edge gateway through the MQTT protocol, wherein the computing power request comprises a computing task type identifier and a priority identifier; after receiving the computing power request, the edge gateway dynamically schedules heterogeneous computing resources of a built-in neural network processing unit (NPU) and a built-in central processing unit (CPU) according to a computing task type identifier and a priority identifier; and the edge gateway executes a corresponding calculation task by using the scheduled calculation resource, feeds back a processing result to the edge terminal initiating the request through an MQTT protocol, and synchronizes own calculation power state information.

Description

Technical Field

[0001] This invention belongs to the fields of edge computing, IoT communication and computing power scheduling technology, and specifically relates to an edge gateway computing power sharing method based on the MQTT protocol. Background Technology

[0002] With the rapid development of IoT technology, edge computing architecture is increasingly being used in industrial automation, smart cities, and smart homes. However, many lightweight edge terminal devices, such as various sensors, cameras, and intelligent monitoring devices, are typically limited by cost, power consumption, and size constraints, and are only equipped with basic data acquisition and simple processing capabilities, making it difficult to support the growing demands for AI inference and complex data processing. In practical deployments, these devices face a severe imbalance in computing resources: some devices have idle computing power, while many terminals lack sufficient computing power, resulting in low overall system efficiency.

[0003] Currently, edge computing power supply mainly relies on two models: one is uploading computing tasks to the cloud for processing, and the other is using dedicated edge computing nodes for independent services. While the cloud-edge collaborative architecture can provide powerful computing power, it faces network bandwidth pressure and transmission latency issues, especially in high-throughput applications (such as video analytics and real-time control). Traditional edge computing nodes typically employ static resource allocation strategies, lacking dynamic perception and response mechanisms for multi-terminal computing power needs, resulting in uneven utilization of computing resources. Heterogeneous computing units (such as NPUs, GPUs, and CPUs) on edge devices are often designed to operate independently, failing to form a collaborative and efficient resource pooling architecture, leading to wasted hardware investment.

[0004] At the communication protocol level, edge computing systems generally use HTTP, CoAP, or proprietary protocols to exchange data between terminals and edge nodes. However, these protocols face significant challenges in computing power sharing scenarios. On the one hand, there is a lack of standardized interface definitions for computing power requests, and the request formats and parameter specifications of devices from different vendors are inconsistent, increasing the complexity of system integration. On the other hand, existing protocol designs do not fully consider computing power status awareness and feedback mechanisms, and terminals cannot dynamically adjust their request strategies based on the real-time load of edge nodes, further exacerbating the uneven distribution of resources. Although lightweight messaging protocols such as MQTT are widely used in IoT device communication, their topic structure and message format design are mostly focused on data acquisition and control command transmission, and have not been specifically optimized for computing power resource sharing scenarios.

[0005] Resource scheduling technology in edge computing environments also faces multiple challenges. Traditional scheduling algorithms are mostly designed for homogeneous computing environments, failing to fully consider the characteristic differences and collaborative optimization of heterogeneous computing units (such as neural network processors and general-purpose processors). In multi-task concurrent scenarios, existing scheduling strategies often lack fine-grained awareness of task type, computational complexity, and real-time requirements, leading to problems such as high-priority task response delays and local overload of computing resources. In addition, the limited memory resources of edge devices are not effectively used for model caching and preloading. Frequent model loading and unloading operations significantly increase task processing latency, making it difficult to meet the demanding real-time requirements of applications such as industrial control and autonomous driving.

[0006] In summary, the field of edge computing urgently needs a computing power sharing architecture that can integrate communication standardization, heterogeneous resource scheduling, and state awareness to overcome the limitations of existing technologies in terms of resource utilization, real-time response, and system compatibility, and achieve efficient collaboration and on-demand allocation of edge computing resources. Summary of the Invention

[0007] To address the shortcomings and deficiencies of existing technologies, this invention provides a method and system for sharing computing power in edge gateways based on the MQTT protocol. The core of this method lies in enabling multiple edge terminals to send computing power requests, including task types and priorities, to the edge gateway through the standardized MQTT communication protocol. Upon receiving a request, the edge gateway does not simply queue the tasks; instead, it performs a fine-grained dynamic scheduling based on task type and priority identifiers, combined with the real-time load status of its built-in Neural Processing Unit (NPU) and Central Processing Unit (CPU). This scheduling process employs a weighted allocation mechanism, prioritizing computationally intensive tasks such as large language model inference to the multi-core collaborative mode of the NPU, allocating tasks such as object detection to the single-core or partial-core mode of the NPU, and directing general-purpose computing tasks to the high-performance cores of the CPU, thereby achieving optimal matching between heterogeneous computing resources and diverse AI tasks.

[0008] To improve overall efficiency and reliability, this invention integrates a series of optimization measures: caching frequently used dedicated format models in the gateway's memory significantly reduces model loading time; periodically synchronizing the gateway's NPU / CPU utilization and task queue status via MQTT topics enables terminals to dynamically adjust request strategies and avoid congestion; and an exception handling mechanism is designed to automatically degrade tasks to CPU execution when NPU execution fails, ensuring task continuity. Furthermore, optimizations to the gateway's operating system process binding and storage partitioning further improve system response speed and stability. This invention achieves deep synergy between communication, scheduling, and hardware optimization, effectively solving the problems of low edge computing resource utilization and high response latency in multi-terminal concurrent scenarios.

[0009] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0010] A method for sharing computing power in edge gateways based on the MQTT protocol, wherein:

[0011] The edge terminal sends a computing power request to the edge gateway via the MQTT protocol. The computing power request includes a computing task type identifier and a priority identifier.

[0012] After receiving the computing power request, the edge gateway dynamically schedules the heterogeneous computing resources of its built-in neural network processing unit (NPU) and central processing unit (CPU) according to the computing task type identifier and priority identifier.

[0013] The edge gateway uses the scheduled computing resources to execute the corresponding computing tasks, feeds back the processing results to the edge terminal that initiated the request via the MQTT protocol, and synchronizes its own computing power status information.

[0014] Furthermore, the edge gateway is a hardware platform based on the RK3588 chip, the NPU contains at least three cores, and the CPU adopts a combination architecture of high-performance cores and energy-efficient cores.

[0015] Furthermore, in the process of dynamically scheduling heterogeneous computing resources, the real-time load status of the NPU and CPU of the RK3588 chip is combined to realize resource allocation through a dynamic load balancing algorithm.

[0016] The dynamic load balancing algorithm adopts a weighted allocation mechanism, with the weight of the pre-set NPU multi-core collaborative working mode being higher than that of the NPU partial core working mode, the NPU single-core working mode, and the CPU. Furthermore, the scheduling weight of computing power requests marked with high priority is higher than that of computing power requests marked with medium and low priority.

[0017] Furthermore, the tasks corresponding to the computing task type identifier include large language model inference tasks, object detection tasks, and general computing tasks;

[0018] The dynamic scheduling of heterogeneous computing resources includes: prioritizing the scheduling of large language model inference tasks to the NPU multi-core collaborative working mode, scheduling object detection tasks to the NPU single-core or partial-core working mode, and scheduling general computing tasks to the high-performance cores of the CPU.

[0019] Furthermore, the edge gateway also performs a model caching step: in the memory resources of the RK3588 chip, a least recently used caching strategy is adopted to cache the processing models corresponding to the large language model inference task and the object detection task;

[0020] When receiving a computing power request, the corresponding processing model is first called from the cache. The processing model is a dedicated format model adapted to the RK3588 chip NPU after format conversion.

[0021] Furthermore, the MQTT protocol implements communication through three predefined topics, which are respectively used by the terminal to publish computing power requests, the gateway to provide feedback on processing results, and the gateway to broadcast computing power status information;

[0022] Both the computing power request and the processing result are encapsulated in a structured data format, which includes a unique terminal identifier, Base64 encoded data content, status identifier, and time-related information.

[0023] Furthermore, the computing power status information is synchronized according to a preset period, and the computing power status information includes NPU utilization, CPU utilization, and remaining task queue length.

[0024] After receiving the computing power status information, the edge terminal dynamically adjusts the computing power request sending strategy: when the gateway computing power utilization rate is higher than a preset high threshold, it delays sending computing power requests with low priority identifiers; when the gateway computing power utilization rate is lower than a preset low threshold, it sends computing power requests in batches.

[0025] Furthermore, it also includes exception handling steps:

[0026] When the execution time of a computing task exceeds the preset timeout threshold in the computing power request, a retry mechanism of a preset number of times is triggered. If the retry fails, a preset error code is returned.

[0027] When the NPU fails to execute a computing task, it automatically demotes the task to the CPU to ensure task continuity.

[0028] Furthermore, it also includes system optimization steps:

[0029] By using the operating system's process binding function, the server process corresponding to the MQTT protocol is bound to a high-performance CPU core;

[0030] The eMMC storage configured on the RK3588 chip is partitioned and optimized, with a dedicated storage area allocated for storing the processing model and request logs, and log compression enabled.

[0031] Regularly clean up expired logs in eMMC storage to ensure that available storage space meets preset requirements.

[0032] And, an edge gateway computing power sharing system based on the MQTT protocol, used to implement the method described above, including an edge terminal and an edge gateway;

[0033] The edge terminal is used to send a computing power request to the edge gateway via the MQTT protocol. The computing power request includes a computing task type identifier and a priority identifier.

[0034] The edge gateway has heterogeneous computing resources of built-in neural network processing unit (NPU) and central processing unit (CPU), and is equipped with a dynamic scheduling module, a communication module and a task execution module.

[0035] The communication module is used to receive the computing power request based on the MQTT protocol, and to feed back the calculation results and synchronize computing power status information.

[0036] The dynamic scheduling module is used to dynamically schedule the heterogeneous computing resources according to the computing task type identifier and priority identifier;

[0037] The task execution module is used to execute the corresponding computing task using the scheduled computing resources, and to feed back the processing result to the edge terminal that initiated the request through the communication module.

[0038] And a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0039] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0040] Compared with the prior art, the present invention and its preferred embodiments have at least the following beneficial effects:

[0041] It effectively solves the compatibility problem of multi-terminal collaborative computing power requests. By building a standardized communication interface through the MQTT protocol, it unifies the interaction format of computing power requests, result feedback and status synchronization, breaks down the protocol barriers of terminals from different manufacturers, and reduces the deployment difficulty and maintenance cost of multi-terminal access and collaboration.

[0042] It significantly improves the computing power utilization efficiency of edge gateways. A dynamic load balancing strategy is designed for the multi-core NPU and CPU big.little core heterogeneous architecture of the RK3588 chip. It allocates computing resources in a differentiated manner based on task type and priority, avoiding the idle computing power in the traditional "one-to-one" service mode and fully tapping the potential of hardware computing power.

[0043] The response speed of computing power sharing has been greatly optimized. By adopting the least recently used caching strategy to cache high-frequency models in memory, the time spent on repeated model loading has been reduced. At the same time, by leveraging the low bandwidth consumption of the MQTT protocol, the communication transmission latency has been reduced, meeting the usage requirements of real-time computing power demand scenarios.

[0044] The system's stability and reliability have been enhanced. Through exception handling mechanisms such as task timeout retries, automatic degradation to CPU inference failures, process core binding, storage partition optimization, and regular log cleanup, problems such as memory leaks and insufficient storage have been avoided, ensuring the stability of the gateway's long-term continuous operation.

[0045] It reduces the hardware cost and network overhead of edge terminals, eliminates the need to configure complex computing hardware for each edge terminal, achieves resource reuse through a computing power sharing model, and reduces the bandwidth consumption of data transmission. It has significant industrial application value and prospects for large-scale promotion, and is suitable for various multi-terminal collaborative computing power needs scenarios such as industrial IoT, smart security, and smart home. Attached Figure Description

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0047] Figure 1 This invention provides an embodiment of the overall architecture diagram illustrating the interaction relationship between the edge terminal, the RK3588 edge gateway (including MQTT Broker, computing power scheduling module, model cache), and the storage module.

[0048] Figure 2 This invention provides a flowchart illustrating the complete process of computing power scheduling, including terminal request sending, gateway reception and parsing, load balancing scheduling, and result feedback.

[0049] Figure 3 This invention provides an interface definition diagram illustrating the three types of MQTT topic formats and data flow directions: request, result, and status. Detailed Implementation

[0050] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0051] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0052] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0053] To address the problems of low computing power utilization, difficulty in multi-terminal collaboration, high communication latency, and underutilization of hardware resources in existing edge gateways, this invention provides a computing power sharing method for RK3588 edge gateways based on MQTT, achieving standardized, efficient, and low-latency computing power sharing. It should be noted that RK3588 is the preferred hardware carrier for implementing this embodiment, but it is not limited to it. Those skilled in the art can replace it with a similar type or edge gateway with corresponding functions according to actual needs.

[0054] This solution is based on an RK3588 edge gateway equipped with 8GB LPDDR4X memory and 64GB eMMC storage, and runs on Ubuntu 20.04. It utilizes an MQTT Broker to build a lightweight computing power sharing communication network, abstracting the RK3588's NPU and CPU computing power as shareable resources and encapsulating them into a standardized MQTT call interface. Edge terminals send computing power requests (including task type, input data, and priority) via the MQTT protocol. The gateway schedules computing resources based on a dynamic load balancing algorithm, calls the RKLLM model or AI inference model to complete the computation task, and then feeds back the results via MQTT. This invention solves the problems of low computing power utilization, difficulty in multi-terminal collaboration, and high communication latency in existing edge gateways. It achieves a computing power sharing response latency of ≤80ms, increases NPU computing power utilization to 92%, supports concurrent requests from 100+ terminals, and is suitable for multi-terminal computing power demand scenarios such as industrial IoT, smart security, and smart homes.

[0055] In one embodiment, the implementation process includes the following steps:

[0056] Step 1: Gateway Environment Deployment: Install MQTT Broker (EclipseMosquitto) on the RK3588 edge gateway, configure TCP communication port - 1883, and enable username and password authentication; deploy RKLLM SDK (adapted for RK3588), OpenCV 4.5.5 and AI inference dependency library, convert the pre-trained model (including RKLLM large model and YOLOv8 inference model) to RKNN format and store it in the / data / models directory of the 64GB eMMC.

[0057] Step 2: Abstraction and Interface Definition of Computing Resources: The computing power of the RK3588's three-core NPU and CPU (4-core A76 + 4-core A55) is abstracted into three types of resources: "Large Model Inference - Computing Power", "Object Detection Computing Power", and "General Computing Power". Standardized interfaces are defined through MQTT topics, including computing power request topic - (device / req / {device_id}), result feedback topic - (gateway / resp / {device_id}), and computing power status topic - (gateway / status).

[0058] Step 3: Sending computing power requests from the terminal: The edge terminal (such as sensors, cameras, smart devices) connects to the MQTT Broker of the gateway through the MQTT client and sends a computing power request in JSON format, which includes the device ID, task type, input data (Base64 encoded), priority (high / medium / - low), and timeout threshold.

[0059] like Figure 3 As shown, the edge terminal publishes a computing power request to the MQTT Broker via the MQTT protocol (topic: `device / req / {device_id}`). The MQTT Broker forwards the request to the RK3588 edge gateway. After the edge gateway completes the task processing, it pushes the computing results to the corresponding terminal via the MQTT Broker (topic: `gateway / resp / {device_id}`). At the same time, the edge gateway continuously collects its own resource status and pushes it to the MQTT Broker via the `gateway / status` topic, which then broadcasts it to each edge terminal.

[0060] Step 4: Gateway Computing Power Scheduling and Task Execution: The gateway receives requests through the MQTT subscription interface, extracts parameters through the request parsing module, and the dynamic load balancing module allocates computing resources based on the current NPU / CPU load and task priority: high-priority tasks preferentially occupy the three NPU cores, while medium- and low-priority tasks are dynamically allocated to the single / dual NPU cores or the CPU; the corresponding model is called (large model inference calls the RKLLM SDK, object detection calls the RKNN model) to execute the calculation, and high-frequency models are cached using 8GB of memory to avoid repeated loading;

[0061] Step 5: Result Feedback and Status Synchronization: The gateway encapsulates the calculation results (including processing status, output data, and time consumption) into JSON format and pushes them to the requesting terminal via the MQTT result feedback topic; at the same time, it synchronizes information such as the gateway's computing power utilization and the number of remaining tasks in real time via the computing power status topic, so that the terminal can dynamically adjust its request strategy.

[0062] Step 6: Exception handling and resource reclamation: When a task times out or inference fails, the gateway triggers a retry mechanism (up to 3 times). If the retry fails, an error code is returned. After the task is completed, computing resources are released, and the cache is reclaimed through the memory management mechanism of Ubuntu 20.04 to ensure resource recycling.

[0063] Preferably, the JSON format of the MQTT interface request in step (2) includes the following fields: "device_id" (string), "task_type" (string, value is "llm_infer" / "object_detect" / "general_calc"), "input_data" (Base64 encoded string), "priority" (string, value is "high" / "medium" / "low"), and "timeout" (integer, unit is ms); the JSON format of the result includes the following fields: "code" (integer, 0 for success), "output_data" (Base64 encoded string), "cost_time" (integer), and "msg" (string).

[0064] As a preferred approach, the dynamic load balancing algorithm in step (4) is based on weighted round-robin, with the following weight allocation rules: NPU triple-core weight 0.6, NPU dual-core weight 0.3, NPU single-core weight 0.1, and CPU weight 0.05. The weight of high-priority tasks is doubled to ensure that urgent tasks are executed first. The weights, as weighting coefficients, are used to adjust the resource availability judgment conditions according to task priority during scheduling, thereby achieving priority scheduling of high-priority tasks.

[0065] As a preferred option, the caching strategy for the 8GB memory in step (4) is as follows: maintain an LRU (Least Recently Used) cache queue, cache a maximum of 5 high-frequency models (including 2 RKLLM models and 3 inference models), the memory occupied by a single model cache is ≤1.5GB, and the model loading time when the cache is hit is ≤100ms.

[0066] As a preferred option, the specific process of calling the RKLLM model in step (4) is as follows: load the librkllmrt.so dynamic library through ctypes, initialize the RKLLMParam parameters (max_context_len=4096, top_p=0.9, temperature=0.8), call the rkllm_run interface to perform inference, and encapsulate and feed back the inference results after Base64 encoding.

[0067] Preferably, a gateway computing power monitoring module is also included, which obtains NPU utilization by reading the / sys / class / rockchip / rknpu / usage node of RK3588, analyzes CPU load by using the top command in Ubuntu 20.04, and updates the monitoring data every 500ms, and synchronizes it through the computing power status topic.

[0068] The following is a more specific application example to further demonstrate and introduce the specific implementation of the solution of the present invention:

[0069] Step 1: Deploy the gateway environment

[0070] Hardware configuration: RK3588 edge gateway (4-core A76 + 4-core A55 + 3-core NPU), equipped with 8GB LPDDR4X memory (3GB reserved for task caching), 64GB eMMC storage (20GB dedicated partition for model files, 10GB for request logs).

[0071] System and software deployment: Install Ubuntu 20.04 LTS arm64 operating system (Linux kernel 5.4.0), install Eclipse Mosquitto 2.0.11 (MQTT Broker) via apt-get, and configure the / etc / mosquitto / mosquitto.conf file to enable authentication: listener 1883 0.0.0.0 allow_anonymous false password_file / etc / mosquitto / pwfile

[0072] Dependency library deployment: Install RKLLM SDK (compatible with RK3588), OpenCV 4.5.5 (with NEON optimization enabled), Python 3.8, paho-mqtt 1.6.1, and compile and install the librkllmrt.so dynamic library to the / usr / lib directory;

[0073] Model deployment: Convert DeepSeek-R1-Distill-Qwen-1.5B_W8A8_RK3588.rkllm (large model) and YOLOv8n.rknn (object detection model) to RK3588 compatible format, store them in the / data / models directory of the 64GB eMMC, and set the file permissions to 755.

[0074] Step 2: Abstraction of computing resources and definition of interfaces

[0075] Computing power resource classification: Based on the characteristics of RK3588 hardware, three types of shareable computing power are abstracted:

[0076] 1. High-performance model inference computing power: Based on a triple-core NPU and RKLLM SDK, it supports text generation and semantic understanding tasks;

[0077] 2. Target detection computing power: Based on NPU single-core / dual-core + YOLOv8n.rknn, it supports target detection in 1080P video frames;

[0078] 3. General computing power: Based on the CPU A76 core, it supports lightweight tasks such as data preprocessing and format conversion;

[0079] MQTT theme design:

[0080] Interface data format:

[0081] Example of a request JSON:

[0082] {

[0083] "device_id": "cam_001",

[0084] "task_type": "object_detect",

[0085] "input_data": "Base64 encoded 1080P video frame data",

[0086] "priority": "high",

[0087] "timeout": 100

[0088] }

[0089] Example of JSON result:

[0090] {

[0091] "code": 0,

[0092] "output_data": "Base64 encoded detection results (target category, coordinates)",

[0093] "cost_time": 75,

[0094] "msg": "success"

[0095] }

[0096] Step 3: Send terminal computing power request

[0097] Terminals send computing power requests via MQTT clients: Edge terminals (such as smart cameras and sensors) connect to the gateway MQTT Broker through the paho-mqtt library and send computing power requests. The core implementation includes: establishing a connection with the gateway MQTTBroker, encoding the input data using Base64, encapsulating the request parameters in a standardized JSON format and publishing them, and subscribing to the result feedback topic to receive the processing results.

[0098] Core code snippet (Python):

[0099] import base64

[0100] import paho.mqtt.client as mqtt

[0101] import json

[0102] import cv2

[0103] import numpy as np

[0104] class MQTTClient:

[0105] def __init__(self, broker_ip, device_id):

[0106] self.broker_ip = broker_ip

[0107] self.device_id = device_id

[0108] self.client = mqtt.Client(client_id=device_id)

[0109] self.client.username_pw_set("mqtt_user", "mqtt_pass")

[0110] self.client.on_connect = self.on_connect

[0111] self.client.on_message = self.on_message

[0112] self.client.connect(broker_ip, 1883, 60)

[0113] self.result = None

[0114] def on_connect(self, client, userdata, flags, rc):

[0115] self.client.subscribe(f"gateway / resp / {self.device_id}")

[0116] def on_message(self, client, userdata, msg):

[0117] self.result = json.loads(msg.payload.decode())

[0118] def send_request(self, task_type, input_data, priority="medium",timeout=100):

[0119] # Input data Base64 encoding

[0120] if isinstance(input_data, np.ndarray):

[0121] _, img_encoded = cv2.imencode('.jpg', input_data)

[0122] input_base64 = base64.b64encode(img_encoded).decode()

[0123] else:

[0124] input_base64 = base64.b64encode(input_data.encode()).decode()

[0125] # Construct and publish the request

[0126] request_data = {

[0127] "device_id": self.device_id,

[0128] "task_type": task_type,

[0129] "input_data": input_base64,

[0130] "priority": priority,

[0131] "timeout": timeout

[0132] }

[0133] self.client.publish(f"device / req / {self.device_id}",json.dumps(request_data))

[0134] self.client.loop_start()

[0135] self.client.loop_stop(timeout=timeout / 1000)

[0136] return self.result

[0137] Step 4: Gateway computing power scheduling and task execution

[0138] like Figure 1 As shown, the edge terminal side includes terminal devices such as smart cameras, sensors, and smart devices. Each device interacts with the RK3588 edge gateway through the MQTT Broker (sending computing power requests and receiving result feedback). The RK3588 edge gateway has a built-in computing power scheduling module, model caching module, storage module, and inference model: the computing power scheduling module is responsible for allocating NPU / CPU computing resources, the storage module is used to cache model files, and the inference model corresponds to different types of tasks (such as large model inference and object detection).

[0139] The gateway, based on Python and multithreading, implements request reception, computing power scheduling, and task execution. It receives terminal requests and schedules computing power, comprising five main modules: model initialization, dynamic load balancing, LRU model caching, task processing, and resource monitoring. The collaborative workflow of each module is as follows:

[0140] 4.1 Global Variables and Model Initialization

[0141] import paho.mqtt.client as mqtt

[0142] import json

[0143] import base64

[0144] import threading

[0145] import time

[0146] import ctypes

[0147] import os

[0148] from collections import deque

[0149] from rknnlite.api import RKNNLite

[0150] # Global core variables

[0151] task_queue = deque() # Task queue

[0152] model_cache = {} # LRU model cache

[0153] cache_max_size = 5 # Maximum number of cache models

[0154] npu_usage = 0.0 # NPU utilization

[0155] cpu_usage = 0.0 # CPU utilization

[0156] # RKLLM large model initialization (adapted to RK3588 NPU triple core)

[0157] class RKLLMInfer:

[0158] def __init__(self, model_path):

[0159] self.rkllm_lib = ctypes.CDLL(' / usr / lib / librkllmrt.so')

[0160] # Define the model parameter structure

[0161] class RKLLMParam(ctypes.Structure):

[0162] _fields_ = [

[0163] ("model_path", ctypes.c_char_p),

[0164] ("max_context_len", ctypes.c_int32),

[0165] ("top_p", ctypes.c_float),

[0166] ("temperature", ctypes.c_float)

[0168] self.rkllm_param = RKLLMParam()

[0169] self.rkllm_param.model_path = bytes(model_path, 'utf-8')

[0170] self.rkllm_param.max_context_len = 4096

[0171] self.rkllm_param.top_p = 0.9

[0172] self.rkllm_param.temperature = 0.8

[0173] # Initialize the model

[0174] self.handle = ctypes.c_void_p()

[0175] self.rkllm_lib.rkllm_init.argtypes = [ctypes.POINTER(ctypes.c_void_p), ctypes.POINTER(RKLLMParam)]

[0176] self.rkllm_lib.rkllm_init(ctypes.byref(self.handle),ctypes.byref(self.rkllm_param))

[0177] def infer(self, input_text):

[0178] input_data = ctypes.c_char_p(input_text.encode('utf-8'))

[0179] output = ctypes.c_char_p() ​

[0180] self.rkllm_lib.rkllm_run(self.handle, input_data,ctypes.byref(output))

[0181] return output.value.decode('utf-8')

[0182] # Initialize YOLOv8 object detection model (adapted to single-core / dual-core NPU)

[0183] class ObjectDetectInfer:

[0184] def __init__(self, model_path):

[0185] self.rknn_lite = RKNNLite()

[0186] self.rknn_lite.load_rknn(model_path)

[0187] self.rknn_lite.init_runtime(runtime_core=RKNNLite.RKNN_NPU_CORE_AUTO)

[0188] def infer(self, frame):

[0189] outputs = self.rknn_lite.inference(inputs=[frame])

[0190] return outputs[0]

[0191] 4.2 Dynamic Load Balancing and Cache Management

[0192] # LRU model cache management (cache up to 5 high-frequency models)

[0193] def add_to_cache(key, model):

[0194] global model_cache

[0195] if len(model_cache) >= cache_max_size:

[0196] # Remove the least recently used model

[0197] lru_key = next(iter(model_cache.keys()))

[0198] del model_cache[lru_key]

[0199] model_cache[key] = model

[0200] # Dynamic load balancing scheduling (core innovation)

[0201] def schedule_task(task):

[0202] global npu_usage, cpu_usage

[0203] task_type = task["task_type"]

[0204] priority = task["priority"]

[0205] # Weight Calculation: High-priority tasks have double the weight.

[0206] weight = 1.0 if priority != "high" else 2.0

[0207] # Large model inference task: Prioritize allocating three NPU cores

[0208] if task_type == "llm_infer" and npu_usage < 70 * weight:

[0209] model_key = "llm_deepseek"

[0210] if model_key not in model_cache:

[0211] model = RKLLMInfer(" / data / models / DeepSeek-R1-Distil-Qwen-1.5B_W8A8_RK3588.rkllm")

[0212] add_to_cache(model_key, model)

[0213] input_data = base64.b64decode(task["input_data"]).decode('utf-8')

[0214] return model_cache[model_key].infer(input_data)

[0215] # Object detection task: Dynamically allocate NPU cores

[0216] elif task_type == "object_detect" and npu_usage < 80 * weight:

[0217] model_key = "yolov8n"

[0218] if model_key not in model_cache:

[0219] model = ObjectDetectInfer(" / data / models / yolov8n.rknn")

[0220] add_to_cache(model_key, model)

[0221] input_data = base64.b64decode(task["input_data"])

[0222] frame = cv2.imdecode(np.frombuffer(input_data, np.uint8),cv2.IMREAD_COLOR)

[0223] return model_cache[model_key].infer(frame)

[0224] # General computing task: Allocate CPU A76 cores

[0225] else:

[0226] return cpu_general_calc(task["input_data"])

[0227] # Implementation of general computing tasks (text description of core logic)

[0228] def cpu_general_calc(input_data):

[0229] # Core logic: Decode input data using Base64, perform lightweight computations such as data preprocessing and format conversion.

[0230] decoded_data = base64.b64decode(input_data).decode('utf-8')

[0231] # The specific calculation logic is omitted here; the actual implementation needs to be expanded according to the task type.

[0232] return {"result": decoded_data, "status": "completed"}

[0233] like Figure 2 As shown, the complete execution flow of a computing power request is as follows: the edge terminal initiates a computing power request, which is transmitted to the MQTT Broker via the MQTT protocol; after receiving the request, the MQTT Broker parses the task information by the request parsing module; the computing power scheduling module allocates resources based on a dynamic load balancing strategy (high-priority tasks are allocated to three NPU cores, medium- and low-priority tasks are allocated to two / one NPU cores, and lightweight tasks are allocated to the CPU), and calls the corresponding inference model or performs general computing; after inference is completed, the result is encapsulated and then fed back to the terminal via the MQTT protocol, and finally the terminal receives the result.

[0234] 4.3 Task Processing and Resource Monitoring

[0235] # Task processing thread

[0236] def process_task():

[0237] while task_queue:

[0238] task_info = task_queue.popleft()

[0239] device_id = task_info["device_id"]

[0240] task = task_info["task"]

[0241] try:

[0242] # Execute the task and calculate the time taken

[0243] start_time = time.time()

[0244] output_data = schedule_task(task)

[0245] cost_time = int((time.time() - start_time) * 1000)

[0246] # Successful encapsulation result

[0247] result = {

[0248] "code": 0,

[0249] "output_data": base64.b64encode(json.dumps(output_data).encode()).decode(),

[0250] "cost_time": cost_time,

[0251] "msg": "success"

[0252] }

[0253] except Exception as e:

[0254] # Encapsulate the exception result

[0255] result = {"code": -1, "output_data": "", "cost_time": 0,"msg": str(e)}

[0256] # Feedback results to the terminal

[0257] client.publish(f"gateway / resp / {device_id}", json.dumps(result))

[0258] # Computing power status monitoring thread (updated every 500ms)

[0259] def monitor_resource():

[0260] global npu_usage, cpu_usage

[0261] while True:

[0262] # Read NPU utilization (RK3588 system node)

[0263] with open(" / sys / class / rockchip / rknpu / usage", "r") as f:

[0264] npu_usage = float(f.read().strip())

[0265] # Read CPU utilization (Ubuntu 20.04 system command)

[0266] cpu_cmd = "top -bn1 | grep 'Cpu(s)' | sed 's / .*, *\\([0-9.]*\\)%* id.* / \\1 / ' | awk '{print 100-$1}'"

[0267] cpu_usage = float(os.popen(cpu_cmd).read().strip())

[0268] # Push computing power status

[0269] status = {

[0270] "npu_usage": round(npu_usage, 2),

[0271] "cpu_usage": round(cpu_usage, 2),

[0272] "task_queue_size": len(task_queue),

[0273] "timestamp": int(time.time() * 1000)

[0274] }

[0275] client.publish("gateway / status", json.dumps(status))

[0276] time.sleep(0.5)

[0277] # MQTT server startup and request reception

[0278] client = mqtt.Client()

[0279] client.username_pw_set("mqtt_user", "mqtt_pass")

[0280] def on_message(client, userdata, msg):

[0281] device_id = msg.topic.split(" / ")[-1]

[0282] request_data = json.loads(msg.payload.decode())

[0283] task_queue.append({"device_id": device_id, "task": request_data})

[0284] threading.Thread(target=process_task).start()

[0285] if __name__ == "__main__":

[0286] client.on_message = on_message

[0287] client.connect("0.0.0.0", 1883, 60)

[0288] client.subscribe("device / req / #") # Subscribe to all terminal requests

[0289] # Start the computing power monitoring thread

[0290] threading.Thread(target=monitor_resource, daemon=True).start()

[0291] client.loop_forever()

[0292] After completing the deployment of the above task scheduling and execution modules, the system operating environment needs to be optimized as follows to further improve performance.

[0293] The above process uses the taskset command in Ubuntu 20.04 to bind the MQTT server process to the CPUA76 core, avoiding process switching overhead; the 64GB eMMC uses the ext4 file system and enables journal compression to reduce disk I / O time.

[0294] Step 5: Result Feedback and Status Synchronization

[0295] Results feedback: The gateway encodes the calculation results in Base64 and pushes them to the corresponding terminal via the results feedback topic. The terminal then decodes and uses the results directly.

[0296] Status synchronization: The gateway pushes the computing power status every 500ms, and the terminal dynamically adjusts its requests based on the status: when the gateway's computing power utilization is >90%, low-priority tasks are delayed in being sent; when the utilization is <30%, tasks can be sent in batches.

[0297] Step 6: Exception Handling and Resource Recovery

[0298] Task timeout handling: The gateway maintains a task timeout timer. When the task execution time exceeds the timeout threshold set by the terminal, a retry is triggered (up to 3 times). If the retry fails, an error code -2 is returned.

[0299] Model inference failure: When NPU inference fails (e.g., model file corruption), it automatically degrades to CPU inference to ensure task continuity;

[0300] Resource reclamation: After the task is completed, memory is reclaimed through Python's gc module, and the rkllm_destroy interface is called to release large model resources to avoid memory leaks; expired logs (more than 7 days) in the 64GB eMMC are cleaned up regularly to ensure that the available storage space is ≥10GB.

[0301] The main innovative designs of the above solutions in the embodiments of the present invention include:

[0302] 1. Construct a lightweight large-model access solution that integrates standardized MQTT communication with RK3588 hardware. This solution addresses multi-terminal compatibility issues through a unified MQTT interface, reducing costs by 30%-40% compared to traditional Socket communication.

[0303] 2. Design a dynamic load balancing strategy adapted to RK3588, based on real-time load allocation of computing resources by NPU / CPU, combined with an 8GB memory model caching mechanism, to reduce the time spent on repeated loading, and improve the NPU computing power utilization rate to over 85%.

[0304] 3. Optimize the RKLLM format for DeepSeek's lightweight large models to fully adapt to the RK3588 NPU's triple-core operating characteristics and support concurrent access from multiple terminals;

[0305] 4. Optimize resource management by leveraging the features of Ubuntu 20.04 system, including process kernel binding, eMMC storage partitioning, and regular log cleanup, to improve the long-term stability of the gateway, achieving 72 consecutive hours without memory leaks or crashes.

[0306] Performance comparison of this invention with existing solutions (based on test of 100 concurrent requests from edge terminals):

[0307] Performance indicators Cloud-based large model access solution Traditional edge large model access scheme Invention Solution Increase Access latency (average) 300ms / time 150ms / time 75ms / time Reduced by 75%-50% NPU computing power utilization - 55% 86% An increase of approximately 56% Concurrency Support No upper limit (depends on bandwidth) 30 terminals 80+ terminals An increase of approximately 167%. Model loading time - 500ms 100ms Reduce by 80% Single terminal communication bandwidth 500Kbps 300Kbps <100Kbps Save 80%-67% 72-hour stability No crashes crashed twice No crashes Significantly improves stability

[0308] The following provides detailed information about a test instance:

[0309] 1: Hardware and System Configuration

[0310] Core hardware: RK3588 edge gateway (4-core A76 + 4-core A55 + 3-core NPU), 8GB LPDDR4X memory, 64GB eMMC storage, 1 RJ45 Gigabit Ethernet port;

[0311] Peripheral configuration: Connects 100 edge terminals (including 50 smart cameras, 30 sensors, and 20 smart devices) to the same local area network via Ethernet;

[0312] Software environment: Ubuntu 20.04 LTS arm64, Eclipse Mosquitto 2.0.11, RKLLM SDK 1.4.0, OpenCV 4.5.5, Python 3.8, paho-mqtt 1.6.1.

[0313] 2: MQTT Broker Configuration and Terminal Access

[0314] Configure MQTT authentication: Execute `mosquitto_passwd -c / etc / mosquitto / pwfile mqtt_user` and set the password `mqtt_pass`;

[0315] Start the MQTT Broker: `systemctl start mosquitto && systemctl enablemosquitto`, and verify port listening using `netstat -tuln | grep 1883`.

[0316] Terminal access: All edge terminals connect to the gateway IP (192.168.1.100) via an MQTT client, subscribe to their own result feedback topics, and send computing power requests.

[0317] 3: Model Deployment and Cache Configuration

[0318] Model conversion: Convert the DeepSeek-R1-Distill-Qwen-1.5B model to RKLLM format using RKNN-Toolkit2 and save it to the / data / models directory;

[0319] Cache configuration: Set the maximum number of model caches to 5, with a maximum memory usage of 1.5GB for a single model. LRU management is implemented through the model_cache dictionary in the code.

[0320] 4: Performance Testing and Verification

[0321] Concurrency test: 100 terminals simultaneously sent target detection requests (1080P video frames). The gateway successfully processed all requests with an average response latency of 75ms and an average NPU utilization of 91.2%.

[0322] Stability test: Run continuously for 72 hours, process more than 100,000 tasks, no memory leaks (memory usage is stable at 4-5GB), no crashes;

[0323] Communication latency test: MQTT data packets were captured using Wireshark, and the average communication latency was calculated to be 18ms, which meets the real-time requirements;

[0324] Storage test: 64GB eMMC continuously stored 7 days of logs, occupying approximately 8GB of space, with ample remaining space, indicating that the log cleanup mechanism is effective.

[0325] The MQTT-based RK3588 edge gateway computing power sharing method provided by this invention can be widely applied in scenarios such as Industrial Internet of Things (ICIoT) (production line defect detection terminals sharing computing power), smart security (multiple cameras sharing target detection computing power), and smart home (smart devices sharing semantic understanding computing power). This method fully leverages the hardware potential of the RK3588 to achieve low-latency, high-utilization, and standardized computing power sharing, significantly reducing the hardware cost and network overhead of edge terminals, and possesses significant industrial application value and prospects for large-scale promotion.

[0326] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0327] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0328] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0329] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0330] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of edge gateway computing power sharing methods based on the MQTT protocol. All equivalent changes and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A method for sharing computing power in edge gateways based on the MQTT protocol, characterized in that: The edge terminal sends a computing power request to the edge gateway via the MQTT protocol. The computing power request includes a computing task type identifier and a priority identifier. After receiving the computing power request, the edge gateway dynamically schedules the heterogeneous computing resources of its built-in neural network processing unit (NPU) and central processing unit (CPU) according to the computing task type identifier and priority identifier. The edge gateway uses the scheduled computing resources to execute the corresponding computing tasks, feeds back the processing results to the edge terminal that initiated the request via the MQTT protocol, and synchronizes its own computing power status information.

2. The edge gateway computing power sharing method based on the MQTT protocol according to claim 1, characterized in that: The edge gateway is a hardware platform based on the RK3588 chip, the NPU contains at least three cores, and the CPU adopts a combination architecture of high-performance cores and energy-efficient cores.

3. The edge gateway computing power sharing method based on the MQTT protocol according to claim 2, characterized in that: In the process of dynamically scheduling heterogeneous computing resources, the real-time load status of the NPU and CPU of the RK3588 chip is also combined to realize resource allocation through a dynamic load balancing algorithm. The dynamic load balancing algorithm adopts a weighted allocation mechanism, with the weight of the pre-set NPU multi-core collaborative working mode being higher than that of the NPU partial core working mode, the NPU single-core working mode, and the CPU. Furthermore, the scheduling weight of computing power requests marked with high priority is higher than that of computing power requests marked with medium and low priority.

4. The edge gateway computing power sharing method based on the MQTT protocol according to claim 2, characterized in that: The tasks corresponding to the computing task type identifier include large language model inference tasks, object detection tasks, and general computing tasks. The dynamic scheduling of heterogeneous computing resources includes: prioritizing the scheduling of large language model inference tasks to the NPU multi-core collaborative working mode, scheduling object detection tasks to the NPU single-core or partial-core working mode, and scheduling general computing tasks to the high-performance cores of the CPU.

5. The edge gateway computing power sharing method based on the MQTT protocol according to claim 4, characterized in that: The edge gateway also performs a model caching step: in the memory resources of the RK3588 chip, a least recently used caching strategy is used to cache the processing models corresponding to the large language model inference task and the object detection task. When receiving a computing power request, the corresponding processing model is first called from the cache. The processing model is a dedicated format model adapted to the RK3588 chip NPU after format conversion.

6. The edge gateway computing power sharing method based on the MQTT protocol according to claim 1, characterized in that: The MQTT protocol enables communication through three predefined topics, which are used for terminals to publish computing power requests, gateways to provide processing results, and gateways to broadcast computing power status information, respectively. Both the computing power request and the processing result are encapsulated in a structured data format, which includes a unique terminal identifier, Base64 encoded data content, status identifier, and time-related information.

7. The edge gateway computing power sharing method based on the MQTT protocol according to claim 1, characterized in that: The computing power status information is synchronized according to a preset period, and the computing power status information includes NPU utilization, CPU utilization, and remaining task queue length. After receiving the computing power status information, the edge terminal dynamically adjusts the computing power request sending strategy: when the gateway computing power utilization rate is higher than a preset high threshold, it delays sending computing power requests with low priority identifiers; when the gateway computing power utilization rate is lower than a preset low threshold, it sends computing power requests in batches.

8. The edge gateway computing power sharing method based on the MQTT protocol according to claim 1, characterized in that: It also includes exception handling steps: When the execution time of a computing task exceeds the preset timeout threshold in the computing power request, a retry mechanism of a preset number of times is triggered. If the retry fails, a preset error code is returned. When the NPU fails to execute a computing task, it automatically demotes the task to the CPU to ensure task continuity.

9. The edge gateway computing power sharing method based on the MQTT protocol according to claim 2, characterized in that: It also includes system optimization steps: By using the operating system's process binding function, the server process corresponding to the MQTT protocol is bound to a high-performance CPU core; The eMMC storage configured on the RK3588 chip is partitioned and optimized, with a dedicated storage area allocated for storing the processing model and request logs, and log compression enabled. Regularly clean up expired logs in eMMC storage to ensure that available storage space meets preset requirements.

10. An edge gateway computing power sharing system based on the MQTT protocol, used to implement the method as described in claims 1-9, characterized in that, Including edge terminals and edge gateways; The edge terminal is used to send a computing power request to the edge gateway via the MQTT protocol. The computing power request includes a computing task type identifier and a priority identifier. The edge gateway has heterogeneous computing resources of built-in neural network processing unit (NPU) and central processing unit (CPU), and is equipped with a dynamic scheduling module, a communication module and a task execution module. The communication module is used to receive the computing power request based on the MQTT protocol, and to feed back the calculation results and synchronize computing power status information. The dynamic scheduling module is used to dynamically schedule the heterogeneous computing resources according to the computing task type identifier and priority identifier; The task execution module is used to execute the corresponding computing task using the scheduled computing resources, and to feed back the processing result to the edge terminal that initiated the request through the communication module.

Images