Data synchronization methods, systems, and storage media based on thread and coroutine scheduling

Abstract

This application provides a data synchronization method, system, and storage medium based on thread and coroutine scheduling. The data synchronization method includes: obtaining an execution request for a data synchronization task; when the data synchronization task meets a first preset condition, requesting a thread to execute the data synchronization task; or, when the data synchronization task meets a second preset condition, requesting a coroutine to execute the data synchronization task. The data synchronization method, system, and storage medium based on thread and coroutine scheduling provided in this application improve the utilization of system resources and the efficiency of data synchronization through flexible scheduling of threads and coroutines.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, specifically to a data synchronization method, system, and storage medium based on thread and coroutine scheduling. Background Technology

[0002] The data synchronization platform is used by all of an enterprise's information systems. It collects data from these systems from one location to the platform through various methods – this is the data acquisition process. Then, it distributes the collected data to different locations through various methods – this is the data distribution process. Both data acquisition and distribution involve a large amount of data transformation and processing logic. This data comes from complex sources, is diverse in type and quantity, and has varying processing logic, often requiring the data synchronization platform to achieve near real-time performance, high performance, and high throughput. The data synchronization platform utilizes numerous thread pools, such as data listening thread pools, data acquisition thread pools, and data processing thread pools. During peak data synchronization periods, a single computer node of the data synchronization platform may simultaneously run hundreds or even thousands of thread pools and threads.

[0003] In conceiving and implementing this application, the inventors discovered at least the following problems: the existing thread pool model of the data synchronization platform leads to a lot of resources being wasted on thread running state switching and thread context switching; too many threads cause lag, blocking, or even crashes, thus making the entire data synchronization platform unusable.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] This application provides a data synchronization method, system, and storage medium based on thread and coroutine scheduling to alleviate the problems of low data synchronization efficiency and low system resource utilization.

[0006] In one aspect, this application provides a data synchronization method based on thread and coroutine scheduling, specifically including:

[0007] Obtain the execution request for the data synchronization task;

[0008] When the data synchronization task meets the first preset condition, a thread is requested to execute the data synchronization task; or, when the data synchronization task meets the second preset condition, a coroutine is requested to execute the data synchronization task.

[0009] Optionally, the data synchronization method based on thread and coroutine scheduling includes at least one of the following before executing the step of obtaining the execution request for the data synchronization task:

[0010] Create a thread pool to accommodate each requested thread, and create a coroutine pool to accommodate each requested coroutine;

[0011] Collect system metrics, including CPU utilization, I / O utilization, number of running threads, and number of running coroutines;

[0012] Collect task metrics, including the historical average CPU utilization and historical average I / O utilization of the task.

[0013] Collect thread metrics, including real-time CPU utilization and real-time I / O utilization of the thread;

[0014] Collect coroutine metrics, including real-time CPU utilization and real-time I / O utilization of the coroutine.

[0015] Optionally, the first preset condition in the data synchronization method based on thread and coroutine scheduling includes at least one of the following:

[0016] The average CPU usage of the task history is greater than or equal to the first threshold.

[0017] The average IO usage of the task history is less than the second threshold;

[0018] The CPU utilization rate is less than the third threshold;

[0019] The IO utilization rate is greater than or equal to the fourth threshold;

[0020] The second preset condition is not met.

[0021] Optionally, the second preset condition in the data synchronization method based on thread and coroutine scheduling includes at least one of the following:

[0022] The average CPU usage of the task history is less than the first threshold;

[0023] The average IO usage rate of the task history is greater than or equal to the second threshold;

[0024] The CPU utilization rate is greater than or equal to the third threshold;

[0025] The IO utilization rate is less than the fourth threshold;

[0026] The first preset condition is not met.

[0027] Optionally, the data synchronization method based on thread and coroutine scheduling includes the following steps after executing the steps of requesting a thread to execute the data synchronization task when the data synchronization task meets a first preset condition, or requesting a coroutine to execute the data synchronization task when the data synchronization task meets a second preset condition:

[0028] In response to the CPU utilization being greater than or equal to the fifth threshold, the system metrics are read;

[0029] Based on the system metrics, when the number of threads is greater than or equal to the sixth threshold and / or the number of coroutines is less than the seventh threshold, the running threads are traversed and the thread metrics are read.

[0030] When the thread metric of at least one thread meets the third preset condition, the at least one thread is switched to a coroutine.

[0031] Optionally, the data synchronization method based on thread and coroutine scheduling includes the step of switching at least one thread to a coroutine when the thread metric of at least one thread meets the third preset condition:

[0032] When the real-time CPU utilization of at least one thread is less than the eighth threshold, the at least one thread is switched to a coroutine; and / or,

[0033] When the real-time I / O utilization of the at least one thread is greater than or equal to the ninth threshold, the at least one thread is switched to a coroutine.

[0034] Optionally, the data synchronization method based on thread and coroutine scheduling includes the following steps after executing the steps of requesting a thread to execute the data synchronization task when the data synchronization task meets a first preset condition, or requesting a coroutine to execute the data synchronization task when the data synchronization task meets a second preset condition:

[0035] In response to the CPU utilization rate being less than the tenth threshold, the system metrics are read;

[0036] According to the system metrics, when the number of threads is less than the eleventh threshold and / or the number of coroutines is greater than or equal to the twelfth threshold, the running coroutines are traversed and the coroutine metrics are read.

[0037] When the coroutine metric of at least one coroutine meets the fourth preset condition, the at least one coroutine is switched to a thread.

[0038] Optionally, the data synchronization method based on thread and coroutine scheduling includes the step of switching at least one coroutine to a thread when the coroutine metric of at least one coroutine meets the fourth preset condition:

[0039] When the real-time CPU utilization of at least one of the coroutines is greater than or equal to the eighth threshold, the at least one coroutine is switched to a thread; and / or,

[0040] When the real-time IO utilization of the at least one coroutine is less than the ninth threshold, the at least one coroutine is switched to a thread.

[0041] On the other hand, this application provides a data synchronization system, specifically,

[0042] The data synchronization system includes interconnected processors and storage media, wherein:

[0043] The storage medium is used to store computer programs;

[0044] The processor is used to execute the computer program to implement the data synchronization method based on thread and coroutine scheduling as described above.

[0045] On the other hand, this application provides a storage medium, specifically, a computer program stored on the storage medium, which, when executed by a processor, implements the data synchronization method based on thread and coroutine scheduling as described above.

[0046] As described above, the data synchronization method, system, and storage medium based on thread and coroutine scheduling provided in this application improve the utilization of system resources and the efficiency of data synchronization through flexible scheduling of threads and coroutines. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0048] Figure 1 This is a flowchart of a data synchronization method based on thread and coroutine scheduling according to an embodiment of this application.

[0049] Figure 2 This is a flowchart of step S10 in a data synchronization method based on thread and coroutine scheduling according to an embodiment of this application.

[0050] Figure 3 This is a structural diagram of a data synchronization system according to an embodiment of this application.

[0051] Figure 4 This is a flowchart illustrating the scheduling of threads and coroutines in a data synchronization system according to an embodiment of this application.

[0052] Figure 5 This is a flowchart illustrating the switching between threads and coroutines in a data synchronization system according to an embodiment of this application.

[0053] The realization of the objectives, functional features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0055] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0056] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0057] First Embodiment

[0058] On one hand, this application provides a data synchronization method based on thread and coroutine scheduling. Figure 1 This is a flowchart of a data synchronization method based on thread and coroutine scheduling according to an embodiment of this application.

[0059] Please see Figure 1 In one embodiment, the data synchronization method based on thread and coroutine scheduling includes:

[0060] S10: Obtain the execution request for the data synchronization task.

[0061] Data synchronization refers to synchronizing data between different systems.

[0062] S20: When the data synchronization task meets the first preset condition, a thread is requested to execute the data synchronization task; or, when the data synchronization task meets the second preset condition, a coroutine is requested to execute the data synchronization task.

[0063] A thread is the smallest unit of computation that an operating system can schedule. Threads are more suitable for CPU-intensive tasks. A coroutine, also known as a microthread, is a more lightweight entity than a thread. Its execution process is similar to a subroutine or a function call without a return value. During the execution of a coroutine, it can be interrupted within the subroutine and then another subroutine can be executed. It can then return to continue execution at an appropriate time. The cost of coroutine switching is much less than that of thread switching, so coroutines are more suitable for I / O-intensive tasks.

[0064] In this embodiment, the data synchronization method based on thread and coroutine scheduling makes reasonable and flexible requests for scheduling of threads and coroutines according to the execution requests of the data synchronization tasks and preset condition information, thereby improving the utilization of system resources and the efficiency of data synchronization.

[0065] Figure 2 This is a flowchart of step S10 in a data synchronization method based on thread and coroutine scheduling according to an embodiment of this application.

[0066] Please see Figure 2 In one embodiment, the data synchronization method based on thread and coroutine scheduling includes at least one of the following before executing step S10: obtaining the execution request for the data synchronization task:

[0067] S11: Create a thread pool to include each requested thread, and create a coroutine pool to include each requested coroutine.

[0068] Optionally, during the execution of data synchronization tasks, a large number of threads and coroutines will be generated. Establishing thread pools and coroutine pools to accommodate all requested threads and coroutines facilitates management. Coroutine pools can also manage other auxiliary functions, such as exception handling, log reporting, and metric reporting, to save resources.

[0069] S12: Collect system metrics, including CPU utilization, I / O utilization, number of running threads, and number of running goroutines.

[0070] System metrics are the performance indicators of a computer system.

[0071] S13: Collect task metrics, including the historical average CPU usage and historical average IO usage of the task.

[0072] Optionally, task metrics can be used to determine whether a task is CPU-intensive or I / O-intensive.

[0073] S14: Collect thread metrics, including real-time CPU utilization and real-time I / O utilization of the thread.

[0074] S15: Collect coroutine metrics, including real-time CPU utilization and real-time I / O utilization of coroutines.

[0075] In this embodiment, the data synchronization method based on thread and coroutine scheduling collects a set of metrics for the aforementioned system, tasks, threads, and coroutines. This not only displays the data synchronization operation status but also allows for reasonable and flexible scheduling of threads and coroutines based on the metric set. It can schedule threads and coroutines for data synchronization tasks ready to be executed, and switch threads and coroutines for currently executing data synchronization tasks, thereby improving the performance and efficiency of data synchronization tasks. Optionally, the metric set can be used to measure the running status of the data synchronization platform's computer nodes and data synchronization tasks; this application does not limit the content of the metric set.

[0076] In one embodiment, the first preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0077] The average CPU usage of the task history is greater than or equal to the first threshold.

[0078] In one embodiment, the first preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0079] The average IO usage rate throughout the task history is less than the second threshold.

[0080] In one embodiment, the first preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0081] CPU utilization is less than the third threshold.

[0082] In one embodiment, the first preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0083] IO utilization is greater than or equal to the fourth threshold.

[0084] In one embodiment, the first preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0085] The second preset condition is not met.

[0086] Optionally, this application does not limit the magnitude of the first, second, third, and fourth thresholds. In the data synchronization method based on thread and coroutine scheduling, the first threshold can be 50%, used to determine the CPU utilization rate of the task's history. The second threshold can be 50%, used to determine the IO utilization rate of the task's history. The third threshold can be 80%, used to determine the CPU utilization rate. The fourth threshold can be 80%, used to determine the IO utilization rate. Taking into account both the performance and efficiency of data synchronization, suitable magnitudes for the first, second, third, and fourth thresholds are selected.

[0087] Optionally, after obtaining the execution request of the data synchronization task, the thread and coroutine scheduling-based data synchronization method determines the task as CPU-intensive if the average historical CPU utilization of the collected metrics is greater than or equal to a first threshold; if the CPU utilization of the system metrics is less than a third threshold, the current CPU utilization is determined to be low; if the average historical IO utilization of the task is less than a second threshold, the task is determined to be IO-intensive; and if the IO utilization is greater than or equal to a fourth threshold, the current IO utilization is determined to be high. In these cases, a thread can be requested to execute the data synchronization task. Optionally, if the second preset condition is not met, a thread can be requested to execute the data synchronization task.

[0088] In one embodiment, the second preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0089] The average CPU usage over the task history is less than the first threshold.

[0090] In one embodiment, the second preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0091] The average IO usage rate throughout the task history is greater than or equal to the second threshold.

[0092] In one embodiment, the second preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0093] CPU utilization is greater than or equal to the third threshold.

[0094] In one embodiment, the second preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0095] IO utilization is less than the fourth threshold.

[0096] In one embodiment, the second preset condition in the data synchronization method based on thread and coroutine scheduling includes:

[0097] The first preset condition is not met.

[0098] Optionally, after obtaining the execution request of the data synchronization task, the thread and coroutine scheduling-based data synchronization method determines whether the task is an IO-intensive task if the average historical IO utilization of the collected metrics is greater than or equal to a second threshold; if the system metric's IO utilization is less than a fourth threshold, the current IO utilization is determined to be low; if the average historical CPU utilization of the task is less than a first threshold, the task is determined to be a CPU-inefficient task; and if the CPU utilization is greater than or equal to a third threshold, the current CPU utilization is determined to be high. In these cases, a coroutine can be requested to execute the data synchronization task. Optionally, if the first preset condition is not met, a coroutine is requested to execute the data synchronization task.

[0099] In one embodiment, the data synchronization method based on thread and coroutine scheduling includes the following steps after executing S20: when the data synchronization task meets a first preset condition, a thread is requested to execute the data synchronization task; or, when the data synchronization task meets a second preset condition, a coroutine is requested to execute the data synchronization task:

[0100] S21: In response to CPU utilization being greater than or equal to the fifth threshold, read system metrics;

[0101] S22: Based on system metrics, when the number of threads is greater than or equal to the sixth threshold and / or the number of coroutines is less than the seventh threshold, traverse the running threads and read the thread metrics.

[0102] S23: When the thread index of at least one thread meets the third preset condition, switch at least one thread to a coroutine.

[0103] Optionally, this application does not limit the values ​​of the fifth, sixth, and seventh thresholds. In the data synchronization method based on thread and coroutine scheduling, the fifth threshold can be 80%, used to determine the CPU utilization rate of the system metric. The sixth threshold can be twice the number of CPU cores, used to determine the number of currently used threads. The seventh threshold can be 300, used to determine the number of currently used coroutines. Taking into account both the performance and efficiency of data synchronization, suitable values ​​for the fifth, sixth, and seventh thresholds are selected.

[0104] In this embodiment, the data synchronization method based on thread and coroutine scheduling, after requesting a thread to execute a data synchronization task, collects and determines that the CPU utilization rate of the current system metrics is greater than or equal to a fifth threshold. When the CPU utilization rate of the system metrics is high, and it is determined that the number of currently used threads is large and / or the number of used coroutines is small, the currently running threads are traversed, thread metrics are read, and threads that meet the conditions are switched to coroutines to reduce the CPU utilization rate of the current system metrics.

[0105] In one embodiment, the data synchronization method based on thread and coroutine scheduling includes the following step in executing S23: when the thread index of at least one thread meets a third preset condition, switching at least one thread to a coroutine:

[0106] When the real-time CPU utilization of at least one thread is less than the eighth threshold, at least one thread will be switched to a coroutine.

[0107] In one embodiment, the data synchronization method based on thread and coroutine scheduling includes the following step in executing S23: when the thread index of at least one thread meets a third preset condition, switching at least one thread to a coroutine:

[0108] When the real-time I / O utilization of at least one thread is greater than or equal to the ninth threshold, at least one thread will be switched to a coroutine.

[0109] Optionally, this application does not limit the size of the eighth and ninth thresholds. In the thread and coroutine scheduling-based data synchronization method, the eighth threshold can be 50%, used to determine the real-time CPU utilization of the thread. The ninth threshold can be 50%, used to determine the real-time I / O utilization of the thread. Taking into account both the performance and efficiency of data synchronization, suitable sizes for the eighth and ninth thresholds are selected.

[0110] In this embodiment, the data synchronization method based on thread and coroutine scheduling determines that a thread does not currently require intensive CPU resource usage when its real-time CPU utilization is less than the eighth threshold, and at this time, the thread can be switched to a coroutine. In another embodiment, a thread is determined to require intensive I / O resource usage when its real-time I / O utilization is greater than or equal to the ninth threshold, and at this time, at least one thread can be switched to a coroutine.

[0111] In one embodiment, the data synchronization method based on thread and coroutine scheduling includes the following steps after executing S20: when the data synchronization task meets a first preset condition, a thread is requested to execute the data synchronization task; or, when the data synchronization task meets a second preset condition, a coroutine is requested to execute the data synchronization task:

[0112] S24: In response to CPU utilization falling below the tenth threshold, read system metrics;

[0113] S25: Based on system metrics, when the number of threads is less than the eleventh threshold and / or the number of coroutines is greater than or equal to the twelfth threshold, traverse the running coroutines and read the coroutine metrics.

[0114] S26: When the coroutine metric of at least one coroutine meets the fourth preset condition, switch at least one coroutine to a thread.

[0115] Optionally, this application does not limit the size of the tenth, eleventh, and twelfth thresholds. In the data synchronization method based on thread and coroutine scheduling, the tenth threshold is used to determine the CPU utilization rate of the system metric, the eleventh threshold is used to determine the number of currently used threads, and the twelfth threshold is used to determine the number of currently used coroutines. Taking into account both the performance and efficiency of data synchronization, suitable sizes for the tenth, eleventh, and twelfth thresholds are selected. In another embodiment, the tenth threshold can be 80%, equal to the fifth threshold; the eleventh threshold can be twice the number of CPU cores, equal to the sixth threshold; and the twelfth threshold can be 300, equal to the seventh threshold, to facilitate timely switching and scheduling of threads and coroutines. Optionally, the tenth threshold can also be set to be less than the fifth threshold, the eleventh threshold less than the sixth threshold, and the twelfth threshold greater than the seventh threshold, thereby retaining a middle range to avoid repeated switching between threads and coroutines due to fluctuations in thread and coroutine usage.

[0116] In this embodiment, the data synchronization method based on thread and coroutine scheduling, after requesting a thread to execute a data synchronization task, collects and determines that the current system CPU utilization is less than the tenth threshold, indicating that the system's CPU utilization is low at this time. Optionally, if it is further determined that the number of currently running threads is low and / or the number of coroutines is high, the running coroutines can be traversed, coroutine indicators can be read, and coroutines that meet the conditions can be switched to threads to reasonably improve CPU resource utilization.

[0117] In one embodiment, the data synchronization method based on thread and coroutine scheduling includes the following step in executing S26: when the coroutine metric of at least one coroutine meets the fourth preset condition, switching at least one coroutine to a thread:

[0118] When the real-time CPU utilization of at least one coroutine is greater than or equal to the eighth threshold, at least one coroutine will be switched to a thread.

[0119] In one embodiment, the data synchronization method based on thread and coroutine scheduling includes the following step in executing S26: when the coroutine metric of at least one coroutine meets the fourth preset condition, switching at least one coroutine to a thread:

[0120] When the real-time I / O utilization of at least one coroutine is less than the ninth threshold, at least one coroutine will be switched to a thread.

[0121] In this embodiment, the data synchronization method based on thread and coroutine scheduling determines that when the real-time CPU utilization of a coroutine is greater than or equal to the eighth threshold, the current coroutine needs to intensively use CPU resources. Therefore, this coroutine can be switched to a thread to reasonably improve CPU resource utilization. In another embodiment, when the real-time I / O utilization of a coroutine is less than the ninth threshold, it can be determined that the current coroutine does not need to intensively use I / O resources. Therefore, this coroutine can be switched to a thread to reasonably improve CPU resource utilization.

[0122] This application's data synchronization method based on thread and coroutine scheduling designs a set of metrics to collect system, task, thread, and coroutine indicators. This not only displays the platform's operational status but also provides data support for thread and coroutine scheduling. When preparing a data synchronization task for execution, the method provides thread and coroutine scheduling services to satisfy CPU-intensive and I / O-intensive tasks respectively, thereby improving the performance and efficiency of the data synchronization task. During data synchronization, the method provides thread and coroutine switching services through performance monitoring and the metric set, thereby improving system resource utilization and availability.

[0123] Second Embodiment

[0124] On the other hand, this application provides a data synchronization system, specifically,

[0125] A data synchronization system comprises interconnected processors and storage media, wherein: the storage media stores computer programs; and the processor executes the computer programs to implement the thread- and coroutine-based data synchronization methods described above.

[0126] Optionally, the data synchronization system may also include: a relational database for persisting business data; a non-relational database for persisting and querying metric data; a task scheduling middleware for cyclically executing data synchronization tasks and auxiliary tasks such as exception handling, log reporting, and metric reporting; and an application server PC for deploying the data synchronization platform.

[0127] Figure 3 This is a structural diagram of a data synchronization system according to an embodiment of this application.

[0128] Please see Figure 3For example, the data synchronization system maintains a unified thread pool and a coroutine pool. Other auxiliary functions, such as exception handling, log reporting, and metric reporting, all use a common coroutine pool to save resources. The data synchronization system platform reports system metrics, such as CPU utilization, IO utilization, and the number of running threads; it also reports task metrics, such as the historical average CPU utilization and IO utilization of tasks; and it reports thread and coroutine metrics, such as real-time CPU utilization and IO utilization of threads and coroutines. The data synchronization system schedules and switches threads and coroutines based on this set of metrics. Data acquisition refers to collecting data from various systems and transmitting it to the data synchronization platform. Data distribution refers to distributing data from the data synchronization platform to various systems. The metric set is the aforementioned set of indicators used to measure the computer's operating status.

[0129] Figure 4 This is a flowchart illustrating the scheduling of threads and coroutines in a data synchronization system according to an embodiment of this application.

[0130] Please see Figure 4 For example, the scheduling steps of threads and coroutines in a data synchronization system include:

[0131] The data synchronization system initiates a data synchronization task. Based on the task's metrics, the system determines whether the task is CPU-intensive; if not, it requests a coroutine. If the task is CPU-intensive, it uses system metrics; if CPU utilization is high, it requests a coroutine; otherwise, it requests a thread. The data synchronization system then executes the task using either the requested thread or coroutine.

[0132] Figure 5 This is a flowchart illustrating the switching between threads and coroutines in a data synchronization system according to an embodiment of this application.

[0133] Please see Figure 5 For example, the switching steps between threads and coroutines in a data synchronization system include:

[0134] The performance monitoring task begins execution. If the system metric of CPU utilization is high, the system metric of thread count is checked. If the system metric count is high, the currently running threads are iterated through, and thread metrics are checked. If a thread is not CPU-intensive, its data is recorded, and the thread is switched to a coroutine, continuing execution until the performance monitoring task ends. If the system metric of CPU utilization is low, the system metric of thread count is checked. If the system metric count is low, the currently running coroutines are iterated through, and coroutine metrics are checked. If a coroutine is CPU-intensive, its data is recorded, and the coroutine is switched to a thread, continuing execution until the performance monitoring task ends.

[0135] In this embodiment, the data synchronization system collects system, task, thread, and coroutine metrics based on a set of indicators. This not only displays the platform's operational status but also provides data for thread and coroutine scheduling. The data synchronization system provides thread and coroutine scheduling services for data synchronization tasks ready to run, catering to both CPU-intensive and I / O-intensive tasks, thereby improving the performance and efficiency of data synchronization tasks. Through performance monitoring and the indicator set, the data synchronization system provides thread and coroutine switching services for ongoing data synchronization tasks, thereby improving system resource utilization and availability.

[0136] Third Embodiment

[0137] On the other hand, this application provides a storage medium, specifically, a computer program stored on the storage medium, which, when executed by a processor, implements the above-described data synchronization method based on thread and coroutine scheduling.

[0138] As described above, the data synchronization method, system, and storage medium based on thread and coroutine scheduling provided in this application utilize platform-collected metrics data to schedule threads and coroutines reasonably and flexibly. This allows for both scheduling threads and coroutines for data synchronization tasks preparing to be executed and switching between threads and coroutines for ongoing data synchronization tasks. While controlling the number of threads on a single computer node, CPU-intensive data synchronization tasks tend to use threads, while I / O-intensive data synchronization tasks tend to use coroutines, thereby improving the utilization of data synchronization platform resources and the efficiency and availability of data synchronization. This solves the problem of existing solutions using a large number of thread pools and threads indiscriminately, which not only consumes significant system resources due to thread context switching but also leads to stuttering, blocking, and crashes due to an excessive number of threads, rendering the entire platform unusable.

[0139] It should be noted that step designations such as S10 and S20 are used in this application for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S20 first and then S10, etc., but these should all be within the protection scope of this application.

[0140] The embodiments of the data synchronization system and storage medium provided in this application may include all the technical features of any of the above-described method embodiments. The extended and explanatory content of the specification is basically the same as that of the embodiments of the above methods, and will not be repeated here.

[0141] This application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to perform the methods described in the various possible implementations above.

[0142] This application also provides a chip, including a memory and a processor. The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a device with the chip installed performs the methods described in the various possible implementations above.

[0143] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0144] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0145] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0146] The units in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0147] In this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions are generally described in detail only when they appear for the first time. When they appear again, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions that are not described in detail later can be referred to their previous relevant detailed descriptions.

[0148] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0149] The technical features of the present application can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0150] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data synchronization method based on thread and coroutine scheduling, characterized in that, include: Obtain the execution request for the data synchronization task; When the data synchronization task meets the first preset condition, a thread is requested to execute the data synchronization task; Alternatively, when the data synchronization task meets the second preset condition, a coroutine is requested to execute the data synchronization task; The step of obtaining the execution request for the data synchronization task includes at least one of the following: Create a thread pool to accommodate each requested thread, and create a coroutine pool to accommodate each requested coroutine; Collect system metrics, including CPU utilization, I / O utilization, number of running threads, and number of running coroutines; Collect task metrics, including the historical average CPU utilization and historical average I / O utilization of the task. Collect thread metrics, including real-time CPU utilization and real-time I / O utilization of the thread; Collect coroutine metrics, including real-time CPU utilization and real-time I / O utilization of the coroutine; When the data synchronization task meets the first preset condition, a thread is requested to execute the data synchronization task; Alternatively, after the step of requesting a coroutine to execute the data synchronization task when the data synchronization task meets the second preset condition, the following steps are included: In response to the CPU utilization being greater than or equal to the fifth threshold, the system metrics are read; Based on the system metrics, when the number of threads is greater than or equal to the sixth threshold and / or the number of coroutines is less than the seventh threshold, the running threads are traversed and the thread metrics are read. When the thread metric of at least one thread meets the third preset condition, the at least one thread is switched to a coroutine; The step of switching at least one thread to a coroutine when the thread metric of at least one thread meets the third preset condition includes: When the real-time CPU utilization of at least one thread is less than the eighth threshold, the at least one thread is switched to a coroutine; and / or, When the real-time I / O utilization of the at least one thread is greater than or equal to the ninth threshold, the at least one thread is switched to a coroutine.

2. The data synchronization method based on thread and coroutine scheduling according to claim 1, characterized in that, The first preset condition includes at least one of the following: The average CPU usage of the task history is greater than or equal to the first threshold. The average IO usage of the task history is less than the second threshold; The CPU utilization rate is less than the third threshold; The IO utilization rate is greater than or equal to the fourth threshold; The second preset condition is not met.

3. The data synchronization method based on thread and coroutine scheduling according to claim 2, characterized in that, The second preset condition includes at least one of the following: The average CPU usage of the task history is less than the first threshold; The average IO usage rate of the task history is greater than or equal to the second threshold; The CPU utilization rate is greater than or equal to the third threshold; The IO utilization rate is less than the fourth threshold; The first preset condition is not met.

4. The data synchronization method based on thread and coroutine scheduling according to any one of claims 2-3, characterized in that, When the data synchronization task meets the first preset condition, a thread is requested to execute the data synchronization task; Alternatively, after the step of requesting a coroutine to execute the data synchronization task when the data synchronization task meets the second preset condition, the following steps are included: In response to the CPU utilization rate being less than the tenth threshold, the system metrics are read; According to the system metrics, when the number of threads is less than the eleventh threshold and / or the number of coroutines is greater than or equal to the twelfth threshold, the running coroutines are traversed and the coroutine metrics are read. When the coroutine metric of at least one coroutine meets the fourth preset condition, the at least one coroutine is switched to a thread.

5. The data synchronization method based on thread and coroutine scheduling according to claim 4, characterized in that, The step of switching at least one coroutine to a thread when the coroutine metric of at least one coroutine meets the fourth preset condition includes: When the real-time CPU utilization of at least one of the coroutines is greater than or equal to the eighth threshold, the at least one coroutine is switched to a thread; and / or, When the real-time IO utilization of the at least one coroutine is less than the ninth threshold, the at least one coroutine is switched to a thread.

6. A data synchronization system, characterized in that, The data synchronization system includes interconnected processors and storage media, wherein: The storage medium is used to store computer programs; The processor is used to execute the computer program to implement the data synchronization method based on thread and coroutine scheduling as described in any one of claims 1-5.

7. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the data synchronization method based on thread and coroutine scheduling as described in any one of claims 1-5.

Images