Checkpoint distributed system processing method applied to cloud edge end

Abstract

The application relates to the technical field of cloud computing and Internet of Things, in particular to a checkpoint distributed system processing method applied to cloud edge and end, which is characterized in that a checkpoint strategy is set, the task state and variables are periodically saved on Redis during normal operation of the equipment, when a fault occurs during task execution, the calculation information on the Redis is sent to another normally-operating equipment, the task can be continuously and correctly executed, the task is finally completed, the loss caused by the fault is reduced, the previously saved recovery information can be used to continue the calculation from the intermediate state, the lost calculation amount is reduced, and the reliability of the cloud system is enhanced.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing and the Internet of Things, and more specifically, to a checkpoint distributed system processing method applied to the cloud-edge-device architecture. Background Technology

[0002] With the continuous increase in data volume and real-time demands from internet users, distributed in-memory computing frameworks based on cloud, edge, and endpoint have become the preferred tool for corporate data operations and scientific research. To improve computing system performance, processor vendors are constantly increasing the number of transistors on individual chips; on the other hand, they are expanding the system's scale by increasing the number of processors per node and the number of computing nodes, leading to a continuously increasing probability of system failures. Cloud computing, as a technology born to meet the needs of the internet age, has received widespread attention. However, in terms of fault handling capabilities, cloud computing technology still has many shortcomings. First, the system architecture of cloud data centers is dynamic and its complexity is constantly increasing. Second, the basic components of cloud computing platforms are diverse, so various failures always occur in the cloud environment. These failures usually interrupt the normal delivery of cloud services and degrade cloud system performance, and in severe cases, may even lead to economic losses for users.

[0003] Spark's existing distributed system uses checkpointing to store RDD computation results on disk and shorten the lineage, which allows for the LCV (Limited Value Delivery) strategy to move RDD data to disk, thereby further freeing up memory. However, checkpointing doesn't always yield better results because it adds a dedicated job for storing RDD data, an operation that is time-consuming due to disk I / O. Flink, based on streaming computation, designed a lightweight asynchronous snapshot algorithm that sets checkpoints at the end of each computation stage and backs up the corresponding data files in the underlying file system. Whenever a task is interrupted and intermediate computation results are lost, the checkpoint data needs to be read from the underlying file system. However, this approach requires transferring large amounts of data between nodes via network connections and reading files from disk, resulting in high operational costs. Summary of the Invention

[0004] The purpose of this invention is to provide a checkpoint distributed system processing method applied to cloud-edge systems to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, one objective of this invention is to provide a checkpoint distributed system processing method applied to cloud-edge environments, the method comprising the following steps:

[0006] Start the device online, obtain device information, establish a heartbeat with the system, and ensure the system is online;

[0007] The user selects the task to be performed, and the Android device obtains the task information and issues the task.

[0008] The cloud-based scheduling module retrieves the tasks or task sets to be executed from the database, processes them, and breaks down each task into multiple subtasks and sends them to the computing device.

[0009] The computing device parses the task information, retrieves the task from the cloud via an HTTP request, performs sub-task calculations, and sends the calculation information back to Redis every 10% of the progress.

[0010] If a device malfunctions, the CheckPoint will send the data stored in Redis for the malfunctioning device, including computation progress information, currently running subtasks, disconnected task information, and algorithm information, to another idle device. If no malfunction occurs, the computation is complete.

[0011] Furthermore, in the above-mentioned checkpoint distributed system processing method applied to cloud-edge-device architecture, the step of the user selecting the task to be executed, and the Android device obtaining the task information and issuing the task includes:

[0012] First, create a new task set and a task ID set. Then, form an unexecuted task queue and a task queue to control the task execution flow. Split a single task in the task queue into multiple sub-tasks and form a sub-task queue.

[0013] The subtask information includes task ID, subtask ID, vehicle ID, calculation function, status, result, and parameters;

[0014] The Android client selects a suitable computing device from the device scheduler to send information about these subtasks, and uses an event queue to record whether the sending is successful or not.

[0015] After a subtask is successfully sent, the status in the subtask information is changed to "received successfully". When a computing device encounters a fault, the progress information of the faulty device is sent to other idle computing devices until success is achieved.

[0016] Furthermore, in the above-described checkpoint distributed system processing method applied to cloud-edge-device architecture, the computing device parses the task information, retrieves the task from the cloud via an HTTP request, and performs sub-task computation on the device, including:

[0017] Set up a queue of tasks to be executed, and then execute the tasks in the queue using RunAllWithName().

[0018] The task execution process is repeated continuously until all tasks in the task queue are completed.

[0019] Furthermore, in the above-described checkpoint distributed system processing method applied to cloud-edge-device architecture, the continuous repetition of task execution until all tasks in the task queue are completed includes:

[0020] Obtain a single task and execute this task using RunWithName();

[0021] The single task is broken down into multiple subtasks, and all online device information is read from Redis using readDeviceInfo() to form a device list;

[0022] The taskMap() function is called to send the set of subtasks extracted from the task to the vehicle Android device via sendAlgorithmInfo(). The success or failure information of each subtask is recorded in the sendEventList event queue.

[0023] Perform secondary scheduling on the failed subtasks in the event queue, and call sendAlgorithmInfo() again to send them to other online, idle and suitable devices.

[0024] If a device fails and goes offline, the progress information of the failed device is retrieved from Redis and sent to other idle devices until the problem is solved, and the calculation is completed.

[0025] Once the device terminal task is completed, the result information is automatically sent to the cloud and recorded in Redis.

[0026] Furthermore, in the above-mentioned checkpoint distributed system processing method applied to cloud-edge devices, the progress information is sent back to Redis. The status and variables of the task are periodically stored in Redis. When a failure occurs during task execution, the computation information in Redis is sent to another normally operating device to ensure that the task can continue to execute correctly.

[0027] Furthermore, in the above-mentioned checkpoint distributed system processing method applied to cloud-edge-device, the step of sending the progress information of the faulty device to other idle computing devices until success is achieved when the computing device encounters a failure includes:

[0028] Subtasks on the faulty device will be retrieved from Redis by the device scheduler, and a new online device with the same number of faulty subtasks will be selected.

[0029] When no idle computing devices are available, all subtasks running under the main task will monitor the online status of computing devices every 10 seconds. When a new computing device comes online and meets the computing conditions, the subtask will be pulled out of Redis and re-participate in the computing.

[0030] Furthermore, in the above-mentioned checkpoint distributed system processing method applied to cloud-edge-device, the Android client selects a suitable computing device from the device scheduler to send information about these subtasks, and records the success or failure of the sending using an event queue, including:

[0031] The subtask expiration listener is triggered every 10 seconds, causing the subtask to time out and retransmit to LBFO. If no computing device is available, the subtask is pushed back into the expiration listener. The expiration listener is triggered every 10 seconds. When a new computing device connects to the network, the subtask will be sent to the new computing device to participate in the computation.

[0032] Furthermore, in the above-mentioned checkpoint distributed system processing method applied to cloud-edge devices, the computation progress information pre-stored in Redis, wherein Redis supports data persistence, can save data in memory to disk, and can be reloaded for use upon restart. The underlying data structure values ​​of Redis can be of types such as string, hash, list, set, and sorted set.

[0033] Furthermore, in the above-mentioned checkpoint distributed system processing method applied to cloud edge, the progress information is sent back to Redis every 10%, and the storage status is monitored in real time by displaying a progress bar on the front end.

[0034] In the technical solution provided by this invention, by setting a Checkpoint strategy, the calculated task status and variables are periodically saved to Redis during normal operation of the device. When a failure occurs during task execution, the calculation information on Redis is sent to another normally operating device to ensure that the task can continue to be executed correctly and eventually complete the task. This avoids starting from scratch, reduces the losses caused by failure, and allows the previously saved recovery information to be used to restart the calculation from the intermediate state, thereby reducing the amount of lost calculation and enhancing the reliability of the cloud system. Attached Figure Description

[0035] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0036] Figure 1This is a schematic flowchart of an embodiment of the checkpoint distributed system processing method applied to cloud-edge devices in this invention.

[0037] Figure 2 This is a distributed system architecture for a checkpoint distributed system processing method applied to cloud-edge devices in this embodiment of the invention;

[0038] Figure 3 This is a schematic diagram illustrating the working principle of the checkpoint distributed system processing method applied to cloud-edge in this embodiment of the invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0041] Example 1

[0042] As a preferred embodiment, such as Figure 1 As shown, a checkpointing distributed system processing method applied to cloud-edge-device environments includes the following steps:

[0043] Start the device online, obtain device information, establish a heartbeat with the system, and ensure the system is online;

[0044] The user selects the task to be performed, and the Android device obtains the task information and issues the task.

[0045] The cloud-based scheduling module retrieves the tasks or task sets to be executed from the database, processes them, and breaks down each task into multiple subtasks and sends them to the devices.

[0046] The computing device parses the task information, retrieves the task from the cloud via an HTTP request, performs sub-task calculations, and sends the progress information back to Redis every 10%.

[0047] If a device malfunctions, the CheckPoint will send the data stored in Redis for the malfunctioning device, including computation progress information, currently running subtasks, disconnected task information, and algorithm information, to another idle device. If no malfunction occurs, the computation is complete.

[0048] like Figure 2 As shown, the second embodiment of the checkpoint distributed system processing method applied to cloud-edge devices in this invention includes:

[0049] First, create a new task set and a task ID set. Then, form an unexecuted task queue and a task queue to control the task execution flow. Split a single task in the task queue into multiple sub-tasks and form a sub-task queue.

[0050] Subtask information includes task ID, subtask ID, vehicle ID, calculation function, status, result, and parameters;

[0051] The Android client selects a suitable computing device from the device scheduler to send information about these subtasks, and uses an event queue to record whether the sending was successful or failed.

[0052] After a subtask is successfully sent, the status in the subtask information is changed to "received successfully". When a computing device encounters a fault, the progress information of the faulty device is sent to other idle computing devices until success is achieved.

[0053] The third embodiment of the checkpoint distributed system processing method applied to cloud-edge devices in this invention includes:

[0054] Set up a queue of tasks to be executed, and then execute the tasks in the queue using RunAllWithName().

[0055] The task execution process is repeated continuously until all tasks in the task queue are completed.

[0056] The fourth embodiment of the checkpoint distributed system processing method applied to cloud-edge devices in this invention includes:

[0057] Obtain a single task and execute this task using RunWithName();

[0058] The single task is broken down into multiple subtasks, and all online device information is read from Redis using readDeviceInfo() to form a device list;

[0059] The taskMap() function is called to send the set of subtasks extracted from the task to the vehicle Android device via sendAlgorithmInfo(). The success or failure information of each subtask is recorded in the sendEventList event queue.

[0060] Perform secondary scheduling on the failed subtasks in the event queue, and call sendAlgorithmInfo() again to send them to other online, idle and suitable devices.

[0061] If a device fails and goes offline, the progress information of the failed device is retrieved from Redis and sent to other idle devices until the problem is solved, and the calculation is completed.

[0062] Once the device terminal task is completed, the result information is automatically sent to the cloud and recorded in Redis.

[0063] The fifth embodiment of the checkpoint distributed system processing method applied to cloud-edge devices in this invention includes:

[0064] The status and variables of the task are periodically saved in Redis. When a failure occurs during task execution, the computation information in Redis is sent to another normally operating device to ensure that the task can continue to execute correctly.

[0065] The sixth embodiment of the checkpoint distributed system processing method applied to cloud-edge devices in this invention includes:

[0066] Subtasks on the faulty device will be retrieved from Redis by the device scheduler, and a new online device with the same number of faulty subtasks will be selected.

[0067] When no idle computing devices are available, all subtasks running under the main task will monitor the online status of computing devices every 10 seconds. When a new computing device comes online and meets the computing conditions, the subtask will be pulled out of Redis and re-participate in the computing.

[0068] The seventh embodiment of the checkpoint distributed system processing method applied to cloud-edge devices in this invention includes:

[0069] The subtask expiration listener is triggered every 10 seconds, causing the subtask to time out and retransmit to LBFO. If no computing device is available, the subtask is pushed back into the expiration listener. The expiration listener is triggered every 10 seconds. When a new computing device connects to the network, the subtask will be sent to the new computing device to participate in the computation.

[0070] The eighth embodiment of the checkpoint distributed system processing method applied to cloud-edge devices in this invention includes:

[0071] Redis supports data persistence, allowing data in memory to be saved to disk and reloaded for use upon restart. The underlying data structure values ​​in Redis can be of types such as strings, hashes, lists, sets, and sorted sets.

[0072] like Figure 3 As shown, the ninth embodiment of the checkpoint distributed system processing method applied to cloud-edge devices in this invention includes:

[0073] Progress information is fed back to Redis in 10% increments, and the storage status is monitored in real time via a progress bar displayed on the front end. The Checkpoint distributed system has a degree of scalability, achieving data storage by overwriting redundant and duplicate data. Using Redis to store Checkpoint files offers high efficiency, improving system reliability and availability. The Checkpoint task period for saving computation progress can be customized by the user.

[0074] Fault tolerance becomes crucial as an emergency failure handling mechanism in cloud systems. Checkpointing is a widely applicable fault tolerance mechanism, often used as the basis for fault-tolerant resource scheduling in cloud computing to provide reliability services to cloud systems. The basic principle of checkpointing is to periodically save the system's task state and variables to reliable storage devices during system operation. When a failure occurs, the previously saved recovery information can be used to restart calculations from an intermediate state, thereby reducing the amount of lost computation and enhancing the reliability of the cloud system.

[0075] Compared to other fault-tolerance technologies, Checkpoint's advantage lies in its ability to handle any type of failure without requiring consideration of the specific fault. In the event of a failure, the system process is simply rolled back to the most recent checkpoint. Because the Checkpoint fault-tolerance model ensures task completion even after a cloud system failure through its rollback mechanism, it is widely used in cloud computing reliability services.

[0076] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0077] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A checkpointing distributed system processing method applied to cloud-edge-device architecture, characterized in that, The checkpoint distributed system processing method applied to cloud-edge includes the following steps: Start the device online, obtain device information, establish a heartbeat with the system, and ensure the system is online; The user selects the task to be performed, and the Android device obtains the task information and issues the task. The cloud-based scheduling module retrieves the tasks or task sets to be executed from the database, processes them, and breaks down each task into multiple subtasks and sends them to the devices. The computing device parses the task information, retrieves the task from the cloud via an HTTP request, performs sub-task calculations, and sends the progress information back to Redis every 10%. If a device malfunctions, the calculation progress information, currently running subtasks, offline task information, and algorithm information of the faulty device, which are pre-stored in Redis, will be sent to another idle device via CheckPoint. If no malfunction occurs, the calculation is complete.

2. The checkpoint distributed system processing method applied to cloud-edge-device systems according to claim 1, characterized in that, The process of the user selecting a task to perform, and the Android client obtaining the task information and issuing the task, includes: First, create a new task set and a task ID set. Then, form an unexecuted task queue and a task queue to control the task execution flow. Split a single task in the task queue into multiple sub-tasks and form a sub-task queue. The subtask information includes task ID, subtask ID, vehicle ID, calculation function, status, result, and parameters; The Android client selects a suitable computing device from the device scheduler to send information about these subtasks, and uses an event queue to record whether the sending is successful or not. After a subtask is successfully sent, the status in the subtask information is changed to "received successfully". When a computing device encounters a fault, the progress information of the faulty device is sent to other idle computing devices until success is achieved.

3. The checkpoint distributed system processing method applied to cloud-edge-device as described in claim 1, characterized in that: The computing device parses the task information, retrieves the task from the cloud via an HTTP request, and performs sub-task calculations, including: Set up a queue of tasks to be executed, and execute the tasks in the queue using RunAllWithName(). The task execution process is repeated continuously until all tasks in the task queue are completed.

4. The checkpoint distributed system processing method applied to cloud-edge-device systems according to claim 3, characterized in that: The process of repeatedly executing tasks continues until all tasks in the task queue have been completed, including: Obtain a single task and execute this task using RunWithName(); The single task is broken down into multiple subtasks, and all online device information is read from Redis using readDeviceInfo() to form a device list; The taskMap() function is called to send the set of subtasks extracted from the task to the vehicle Android device via sendAlgorithmInfo(). The success or failure information of each subtask is recorded in the sendEventList event queue. Perform secondary scheduling on the failed subtasks in the event queue, and call sendAlgorithmInfo() again to send them to other online, idle and suitable devices. If a device fails and goes offline, the progress information of the failed device is retrieved from Redis and sent to other idle devices until the task is completed. Once the device terminal task is completed, the result information is automatically sent to the cloud and recorded in Redis.

5. The checkpoint distributed system processing method applied to cloud-edge-device as described in claim 1, characterized in that, The progress information is sent back to Redis, where the status and variables of the task are periodically saved. When a failure occurs during task execution, the computation information in Redis is sent to another normally operating device to ensure that the task can continue to execute correctly.

6. The checkpoint distributed system processing method applied to cloud-edge-device systems according to claim 2, characterized in that, The step of sending the progress information of the faulty computing device to other idle computing devices until success is achieved when the computing device encounters a failure includes: Subtasks on the faulty device will be retrieved from Redis by the device scheduler, and a new online device with the same number of faulty subtasks will be selected. When no idle computing devices are available, all subtasks running under the main task will monitor the online status of computing devices every 10 seconds. When a new computing device comes online and meets the computing conditions, the subtask will be pulled out of Redis and re-participate in the computing.

7. The checkpoint distributed system processing method applied to cloud-edge-device architecture according to claim 2, characterized in that, The Android client selects a suitable computing device from the device scheduler to send information about these subtasks, and records the success or failure of the sending using an event queue, including: The subtask expiration listener is triggered every 10 seconds, causing the subtask to time out and retransmit to LBFO. If no computing device is available, the subtask is pushed back into the expiration listener. The expiration listener is triggered every 10 seconds. When a new computing device connects to the network, the subtask will be sent to the new computing device to participate in the computation.

8. The checkpoint distributed system processing method applied to cloud-edge-device according to claim 1, characterized in that, The computation progress information is pre-stored in Redis. Redis supports data persistence, saving data in memory to disk, which can be reloaded and used upon restart. The underlying data structure values ​​of Redis are strings, hashes, lists, sets, and sorted sets.

9. The checkpoint distributed system processing method applied to cloud-edge-device systems according to claim 1, characterized in that, The progress information is sent back to Redis in 10% increments, and the storage status is monitored in real time by displaying a progress bar on the front end.

Images