Data processing method, device, system, medium and program product

Abstract

The present application provides a data processing method, device, system, medium and program product. The method includes: a time slice generation thread detects whether a first token representing that a data producer thread group has completed the production of a data production task corresponding to the Q-1th round, and a second token representing that a data consumer thread group has completed the consumption of a data consumption task corresponding to the Q-P round. If so, the time slice generation thread allocates a time slice corresponding to the data production task of the Qth round to the data producer thread group based on the time point corresponding to the data to be processed and the time slice division parameter. The method of the present application replaces the sleep waiting mechanism with a token notification mechanism to realize the advancement and scheduling of data production tasks, avoids invalid waiting caused by a fixed sleep time in the sleep waiting mechanism, improves the efficiency of data producer thread scheduling, and significantly enhances the overall data processing capability of the system.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a data processing method, device, system, medium and program product. Background Art

[0002] The data processing system can use the producer-consumer thread model combined with the time slice scheduling algorithm to process data tasks. Among them, the producer-consumer thread model is a multi-threaded model that allows multiple threads to be created in one program, and each thread can independently execute different tasks. The time slice scheduling algorithm refers to dividing the processing time of the data task into multiple time slices, allocating each time slice to a data producer thread, and allowing multiple data producer threads to concurrently execute data production tasks according to the allocated time slices. The data producer thread group can produce a large amount of data at the same time, thereby improving the speed and efficiency of data processing.

[0003] The above data processing methods usually rely on a thread periodic sleep mechanism to ensure the reliability and integrity of data task processing. However, the thread periodic sleep mechanism will cause resource waste and response delay of the data processing system, thereby reducing the processing efficiency of data tasks. Summary of the invention

[0004] The present application provides a data processing method, device, system, medium and program product to solve the problem that the thread periodic sleep mechanism brings invalid waiting time to the data producer thread, thereby causing the overall performance and efficiency of the system to decrease.

[0005] On the one hand, the present application provides a data processing method, which is applied to a data processing system, wherein the data processing system runs a time slice generation thread, a data producer thread group, and a data consumer thread group; the method comprises:

[0006] The time slice generation thread detects whether a first token representing that the data producer thread group has completed the production of the data production task corresponding to the Q-1th round, and a second token representing that the data consumer thread group has completed the consumption of the data consumption task corresponding to the QP round, are obtained; the Q and the P are both integers greater than or equal to 1;

[0007] If so, the time slice generation thread allocates the time slice corresponding to the Qth round of data production tasks to the data producer thread group based on the time point corresponding to the to-be-processed data and the time slice division parameters.

[0008] Optionally, the time slice division parameters include: the number of time slices X, and the maximum time slice length Y; the time slice generation thread allocates the time slice corresponding to the Qth round of data production tasks to the data producer thread group based on the time point corresponding to the data to be processed and the time slice division parameters, including:

[0009] The time slice generation thread determines the length of the time slice of the Qth round according to the relationship between the interval between the time point and the current time and the first value, and X; the first value is greater than or equal to the product of X and Y;

[0010] The time slice generation thread obtains X time slices corresponding to the data production task of the Qth round based on the length of the time slice of the Qth round and the time point corresponding to the data to be processed;

[0011] The time slice generation thread writes X time slices into the time slice queue.

[0012] Optionally, when the interval between the time point and the current time is greater than a first value, the length of the time slice corresponding to the Qth round of data production tasks is equal to the Y.

[0013] Optionally, when the interval between the time point and the current time is less than or equal to the first value, the length of the time slice corresponding to the Qth round of data production tasks is equal to the quotient of Y and X.

[0014] Optionally, the data producer thread in the data producer thread group performs the following operations:

[0015] Enter the awakening state;

[0016] Obtaining a corresponding time slice from the time slice queue and executing a data production task for the time slice;

[0017] Add 1 to the number of completed time slices corresponding to the Qth round to obtain an updated number of completed time slices;

[0018] If the updated number of completed time slices is equal to X, reporting to the time slice generation thread a first token indicating that the data producer thread group has completed the data production task corresponding to the Qth round;

[0019] If the updated number of completed time slices is less than X, new time slices continue to be obtained from the time slice queue.

[0020] Optionally, the data processing system is provided with P data queues, each data queue is used to store data packets generated by a round of data consumption tasks; the data consumption task of executing the time slice includes:

[0021] Produce data of the time slice from the data source to obtain the data packet of the time slice;

[0022] The data packets of this time slice are stored in the data queue corresponding to the Qth round.

[0023] Optionally, the data consumer threads in the data consumer thread group perform the following operations:

[0024] Enter the awakening state;

[0025] Obtain a corresponding data packet from the data queue corresponding to the Qth round, and execute a data consumption task for the data packet;

[0026] If there are new data packets in the data queue, continue to obtain new data packets from the data queue corresponding to the Qth round;

[0027] If there is no new data packet in the data queue, and the number of data consumer threads that have completed the current work is equal to N, then a second token representing that the data consumer thread group has completed the consumption of the data consumption task corresponding to the Qth round is reported to the time slice generation thread;

[0028] If there is no new data packet in the data queue and the number of data consumer threads that have completed the current work is less than N, it enters the waiting wake-up state.

[0029] Optionally, the method further comprises:

[0030] After allocating the time slice corresponding to the data production task of the Qth round, if the time slice generation thread has not obtained the first token indicating that the data producer thread group has completed the data production task corresponding to the Qth round when the first preset time is reached, a prompt message indicating that the data production of the Qth round is wrong is output.

[0031] Optionally, the method further comprises:

[0032] After allocating the time slice corresponding to the data production task of the Qth round, if the time slice generation thread has not obtained the second token representing that the data consumer thread group has completed the data consumption task corresponding to the Qth round when reaching the second preset time length, a prompt message representing that the data processing of the Qth round has an error is output.

[0033] Optionally, the data processing system further runs a management thread, and the method further includes:

[0034] The management thread performs an initialization operation based on the data processing task to set the value of one or more of M, N, X, Y, and P.

[0035] Optionally, the method further comprises:

[0036] The management thread generates an initial first token and an initial second token; the initial first token and the second token are used to trigger the time slice generation thread to allocate the time slice corresponding to the first round of data production tasks;

[0037] The management thread sends the initial first token and the initial second token to the time slice generation thread.

[0038] In a second aspect, the present application provides an electronic device, the electronic device comprising: a processor, and a memory communicatively connected to the processor;

[0039] The memory stores computer-executable instructions;

[0040] The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of the first aspects of claim 1.

[0041] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0042] In a fourth aspect, the present application provides a computer program product, including a computer program, which implements the method described in any one of the first aspects when executed by a processor.

[0043] In a fifth aspect, the present application provides a data processing system, wherein the data processing system runs a time slice generation thread, a data producer thread group, and a data consumer thread group;

[0044] The data producer thread in the data producer thread group is used to execute data production tasks;

[0045] The data consumer threads in the data consumer thread group are used to execute data consumption tasks on the data produced by the data producer thread group;

[0046] The time slice generation thread is used to:

[0047] Detect whether a first token representing that the data producer thread group has completed the production of the data production task corresponding to the Q-1th round, and a second token representing that the data consumer thread group has completed the consumption of the data consumption task corresponding to the QP round are obtained; the Q and the P are both integers greater than or equal to 1;

[0048] If so, the time slice generation thread allocates the time slice corresponding to the Qth round of data production tasks to the data producer thread group based on the time point corresponding to the to-be-processed data and the time slice division parameters.

[0049] The data processing method, device, system, medium and program product provided by the present application replace the sleep waiting mechanism with a token notification mechanism to realize the advancement and scheduling of data production tasks, avoiding the invalid waiting caused by the fixed sleep time in the sleep waiting mechanism, thereby reducing the idle time of the data producer thread group, improving the efficiency of the data producer thread scheduling, and significantly enhancing the overall data processing capability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1 A schematic diagram of a data processing method of a data processing system;

[0052] Figure 2 It is a schematic diagram of a data processing flow based on a producer-consumer thread model;

[0053] Figure 3 A schematic diagram of a data processing method of a data processing system provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of a data processing method provided in an embodiment of the present application;

[0055] Figure 5 A flowchart of a method for allocating time slices corresponding to data production tasks by time slice generation threads provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a process for a data producer thread group to perform a data production task is provided for an embodiment of the present application;

[0057] Figure 7 A schematic diagram of a process for a data consumer thread group to perform a data consumption task is provided for an embodiment of the present application;

[0058] Figure 8 A schematic diagram of a process for implementing data processing based on a data processing system provided in an embodiment of the present application;

[0059] Fig. 9 A flowchart of a data processing method for implementing time slice round advancement provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0061] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0063] The following uses a data processing system as an example to explain the terms involved in this application:

[0064] Multithreading model: is a programming paradigm that allows data processing systems to create and manage multiple concurrently executed threads. Each thread can be regarded as an independent stream of data processing task execution and can share the memory space and resources of the data processing system. At the same time, appropriate synchronization mechanisms are required to coordinate access to shared data to ensure data consistency and correct communication between threads, thereby improving the parallelism and responsiveness of the data processing system.

[0065] Concurrency: refers to the fact that in a data processing system, multiple threads can execute multiple tasks at the same time. Each thread can execute different tasks, and there will be no data conflicts between threads.

[0066] Producer-Consumer Model: It is a multi-threaded model commonly used in data processing systems. It decouples the data producer thread group and the consumer thread group by adding a shared buffer, namely a data queue, and uses a synchronization mechanism to coordinate their access to the shared buffer data queue to ensure data consistency and thread safety, thereby achieving efficient data processing and resource utilization.

[0067] Decoupling of the producer-consumer model: In the producer-consumer model, the data producer thread group and the data consumer thread group exchange data indirectly through a shared buffer instead of communicating directly, so that the data producer thread group and the data consumer thread group can independently perform their respective tasks without being affected by the implementation details of the other party.

[0068] Synchronization mechanism: refers to coordinating the access of multiple threads to shared resources through various methods and technologies, such as mutex locks, conditional variables, read-write locks, etc. These mechanisms ensure that the access to shared data is orderly and controllable in a multi-threaded environment, prevent data inconsistency, race conditions, deadlocks and other problems, and maintain data integrity and system correctness.

[0069] Mutex: A basic synchronization mechanism used to protect shared buffer data and ensure that only one thread can access the area at any time. When a thread acquires a mutex, other threads trying to acquire the same mutex will be blocked until the lock is released.

[0070] Time slice: The time allocated by the data processing system to the data producer thread to process data production tasks.

[0071] Time slice scheduling algorithm: By dividing the processing time of the data task into multiple time slices, each time slice is allocated to a data producer thread, and multiple data producer threads are allowed to concurrently execute data production tasks according to the allocated time slices.

[0072] Round: refers to a complete data processing cycle. The data processing system can divide the data processing task into multiple stages or batches for execution. In the producer-consumer thread model, the beginning of a round marks that the data producer thread group starts the data production task, and the end of a round marks that all consumer threads have completed processing the current batch of data.

[0073] Thread periodic sleep mechanism: After completing a certain data production task, the data producer thread will actively enter the sleep state according to the pre-set sleep time. When the sleep time is reached, the synchronization mechanism is used to obtain information on whether the data consumer thread has completed data processing. After the data consumer thread completes the data processing task, it executes the subsequent data production task.

[0074] Blocked state: a waiting state of a thread, which suspends the execution of the thread until a specific wake-up condition is met.

[0075] Atomic operation: When a data producer thread queries a global variable, it ensures that the operation is not interrupted during execution, thereby preventing other threads from modifying the variable at the same time and ensuring the integrity and consistency of the operation.

[0076] A data processing system refers to a system that can perform data production and data processing tasks. In a data processing system, in order to meet the task requirements of various types of data, the producer-consumer thread model can be used to process data in batches or in real time in chronological order.

[0077] The producer-consumer thread model can be combined with a time slice scheduling algorithm to allocate time slices to each data producer thread, allowing them to concurrently execute data production tasks within the allocated time slices, so that the data production author thread group can process a large amount of data at the same time, improving the speed and efficiency of data processing.

[0078] This method usually relies on the thread periodic sleep mechanism to ensure the reliability and integrity of the current round of data processing. That is, after completing the data production task of this round, the data producer thread actively enters the sleep state according to the pre-set sleep time. When the sleep time is reached, the synchronization mechanism is used to obtain information on whether the data consumer thread has completed data processing. After the data consumer thread completes all data processing tasks of this round, it indicates that the data processing of this round is completed and the next round of data tasks can be executed.

[0079] Figure 1 A schematic diagram of a data processing method of a data processing system is shown in FIG. Figure 1 As shown, the data processing system runs a management thread, a data producer thread group, and a data consumer thread group to implement data processing, and resource sharing is achieved between the thread groups through data queues. Figure 2 A schematic diagram of a data processing flow based on a producer-consumer thread model.

[0080] Combine the following Figure 1 and Figure 2 Briefly describe how the data processing system runs management threads, data producer thread groups, and data consumer thread groups to process data:

[0081] (1) Management thread: When a data processing task starts, the management thread is responsible for starting the data producer thread group and the data consumer thread group, and configuring the startup parameters of the data producer thread group. The startup parameters include the time slice length of the data processing task and the time slice information.

[0082] The time slice information is defined as a global variable stored in the static storage area for use by the data producer thread group. Each data producer thread reads or updates the time slice information according to its current task status. The time slice length is a fixed time length and will not change in this round of data processing tasks.

[0083] The management thread may be, for example, a main thread or a control thread, or any thread that can start a data producer thread group and a data consumer thread group and configure startup parameters of the data producer thread group.

[0084] (2) Data producer thread group: It is composed of multiple data producer threads. Each data producer thread is decoupled from each other, and each thread is responsible for independent data production tasks. The data producer thread group can be, for example, a data mining thread group or a data collection thread group, any thread group used to perform data production tasks from a data source.

[0085] After the data producer thread group is started, the data producer thread will enter a blocked state. After obtaining the time slice information, the data producer thread is awakened and starts to execute the data production task of the time slice.

[0086] The data producer thread accesses the data source to obtain available data to perform data production tasks. The data producer thread packages the produced data into data packets and puts them into the data queue.

[0087] Among them, the data source can be, for example, data input by the user in real time, data in an external system, data in a database, real-time data stream, or any other data that needs to be processed. The embodiment of the present application does not limit the data source.

[0088] After the data producer thread atomically reads the time slice information from the static storage area, it updates the time slice information stored in the static storage area so that other data producer threads can determine the next time slice to be processed based on the new time slice information stored in the static storage area.

[0089] Exemplarily, two data production threads in the data producer thread group are used as an example for hypothetical explanation. When the latest time slice information obtained by data producer thread 1 is 0 seconds and the time slice length is 2 seconds, data producer thread 1 will start from 0 seconds and perform data production tasks according to the time slice length of 2 seconds. After obtaining the latest time slice information, data producer thread 1 updates the time slice information to 3 seconds. Data producer thread 3 will obtain the updated time slice information of 3 seconds. Data producer thread 3 will start from the 3rd second and perform data production tasks according to the time slice length of 2 seconds.

[0090] like Figure 2 As shown in a, b and c, after thread 1 in the data producer thread group obtains the time slice information, it updates the time slice information, and thread 2 obtains the updated time slice information. Thread 1 obtains data from the data source according to the time slice information, and the data producer thread puts the produced data packet 1 into the data queue.

[0091] (3) Data consumer thread group: It is composed of multiple data consumer threads. Each data consumer thread is decoupled from each other, and each thread is responsible for independent data processing tasks. The data consumer thread group can be, for example, a data processing thread group or a data analysis thread group, any thread group that can process the produced data.

[0092] After the data consumer thread group is started, the data consumer thread enters a blocked state until there is an available data packet in the data queue. The data consumer threads concurrently obtain data packets from the data queue for processing. Each data consumer thread obtains one data packet at a time for processing.

[0093] like Figure 2 As shown in c and d, thread 1 of the data consumer thread group obtains data packet 1 from the data queue for data processing.

[0094] (4) Data queue: used to store data packets generated by the data producer thread. Data packets generated by the data producer thread are stored in the data queue in the order in which they were created.

[0095] In the data processing system, the data producer thread group can realize batch data production or real-time data production. Batch data production and real-time data production can be set according to the data tasks to be processed.

[0096] When the time point corresponding to the data to be processed is far away from the current time, it means that the amount of unprocessed data accumulated by the data source is relatively large, and the system can enter the batch data production state. Multiple data producer threads in each round of data producer thread group can concurrently obtain multiple time slice information to execute data production tasks at the same time. Batch data production tasks are realized, so that a large amount of data production work can be completed in batches and quickly.

[0097] When the time point corresponding to the data to be processed is close to the current time, the amount of unprocessed data accumulated by the data source is relatively small. Therefore, the system can enter the real-time data production state. The data producer thread group of each round can only obtain one time slice information in real time, so there is only one data producer thread to perform data production tasks in one cycle. Ensure the timeliness of data production work.

[0098] The above data processing method improves the efficiency of data processing to a certain extent. In order to ensure data integrity, the data processing system adopts a sleep waiting mechanism. After completing a round of data production tasks, the data producer thread actively enters the sleep state according to the pre-set sleep time. When the sleep time is reached, the synchronization mechanism is used to obtain information on whether the data consumer thread has completed data processing. After the data consumer thread completes the data processing task, the next round of data production tasks is executed. Since the sleep time of the data producer thread is a pre-set fixed time, even if the data consumption task of the data consumer thread has been completed in advance, the data producer thread group cannot immediately proceed to the next step, which increases the invalid waiting time of the data producer thread. This method causes a decline in the overall performance and efficiency of the system to a certain extent.

[0099] In view of this, an embodiment of the present application proposes a data processing method, which uses a token that can characterize whether the data producer thread group has completed the corresponding round of data production tasks, and a token notification mechanism that can characterize whether the data consumer thread group has completed the corresponding round of data consumption tasks to replace the sleep waiting mechanism of the data producer thread group, so as to realize the advancement and scheduling of data production tasks, avoid the invalid waiting caused by the fixed sleep time in the sleep waiting mechanism, thereby reducing the idle time of the data producer thread group, improving the efficiency of the data producer thread scheduling, and significantly enhancing the overall data processing capability of the system.

[0100] The following is a detailed description of the method provided in the embodiments of the present application. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0101] Figure 3 A schematic diagram of a data processing method of a data processing system provided in an embodiment of the present application, such as Figure 3 As shown, the data processing system runs a time slice production thread, a data producer thread group, and a data consumer thread group.

[0102] The time slice generation thread is responsible for detecting whether a first token representing that the data producer thread group has completed the production of the data production task corresponding to the Q-1 round and a second token representing that the data consumer thread group has completed the consumption of the data consumption task corresponding to the QP round are obtained.

[0103] The time slice generation thread is also responsible for allocating the time slice corresponding to the data production task of the Qth round to the data producer thread group based on the time point corresponding to the data to be processed and the time slice division parameters.

[0104] Exemplarily, the above time slice division parameters may include: the number of time slices X, and the maximum time slice length Y. The time slice generation thread may allocate the time slice corresponding to the data production task of the Qth round to the data producer thread group based on the time point corresponding to the data to be processed in this round and the time slice division parameters.

[0105] Exemplarily, the time slices may be two timestamps, such as [start time, end time], and multiple time slices generated in each round are continuous, and time slices generated in multiple rounds are continuous.

[0106] For example, the data processing system may also be provided with a time slice queue, and the time slice queue may be used to store the time slices generated by the time slice generation thread.

[0107] Exemplarily, the data producer thread in the data producer thread group obtains the corresponding time slice from the time slice queue and executes the data generation task of the time slice, and adds 1 to the number of completed time slices corresponding to the Qth round to obtain the updated number of completed time slices.

[0108] Exemplarily, if the number of time slices completed by the data producer thread group is equal to X, a first token indicating that the data producer thread group has completed the data production task corresponding to the Qth round is reported to the time slice generation thread.

[0109] For another example, the data processing system may also run a management thread.

[0110] Exemplarily, the management thread may be used to perform initialization operations, for example, to set the value of one or more of M, N, X, Y, and P. The time slice generation thread, the data producer thread group, and the data consumer thread group are started.

[0111] For another example, the management thread is further used to generate an initial first token and an initial second token. The initial first token and the second token can be used to trigger the time slice generation thread to allocate the time slice corresponding to the first round of data production tasks.

[0112] For another example, the time slice generation thread obtains an initial first token and an initial second token.

[0113] It should be noted that the time slice generation module adopts a blocking token notification mechanism and can store at most one first token and P second tokens. When the number of stored first tokens and second tokens reaches the upper limit, the data producer thread and the data consumer thread cannot continue to send the first token and the second token.

[0114] Exemplarily, the data processing system may also be provided with a data queue responsible for storing data packets generated by the data processing tasks.

[0115] Exemplarily, in the data processing task of round Q, the data consumer thread in the data consumer thread group obtains the corresponding data packet from the data queue corresponding to the round Q, and executes the data consumption task of the data packet.

[0116] For another example, when the number of data consumer threads that have completed the current work in the data consumer thread group is equal to N, a second token representing that the data consumer thread group has completed the consumption of the data consumption task corresponding to the Qth round is reported to the time slice generation thread.

[0117] It should be noted that the above Figure 3The data processing method of the data processing system shown is only an illustration, and the present application does not limit the threads running in the data processing system, the naming of each thread, and the coordinated operation mode between the threads. For example, in some embodiments, the above-mentioned data producer thread can also be called a sub-model of the data producer thread group, and the data consumer thread can be called a sub-model of the data consumer thread group.

[0118] In one embodiment, the data processing system can be deployed entirely in a cloud environment. For example, if the computing resources included in the cloud environment are servers running virtual machines, the data processing system can be independently deployed on a server or virtual machine in the cloud environment, or the data processing system can be distributedly deployed on multiple servers in the cloud environment, or distributedly deployed on multiple virtual machines in the cloud environment, or distributedly deployed on servers and virtual machines in the cloud environment.

[0119] The data processing system provided by the embodiment of the present application can also be deployed in different environments in a distributed manner. The data processing system provided by the present application can be logically divided into multiple parts, each of which has different functions. The various parts in the data processing system can be deployed in any two or three of the terminal computing device (located on the user side), the edge environment, and the cloud environment. The various parts of the data processing system deployed in different environments or devices cooperate to realize the function of data processing. It should be understood that the embodiment of the present application does not restrictively divide which parts of the data processing system are deployed in what environment. In actual application, it can be adaptively deployed according to the computing power of the terminal computing device, the resource occupancy of the edge environment and the cloud environment, or the specific application requirements.

[0120] The data processing system may also be deployed separately on a computing device in any environment (for example, deployed separately on an edge server in an edge environment).

[0121] The following is a detailed description of the technical solution of how to use the data processing system to process data in the embodiments of the present application in combination with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0122] In this embodiment, when the data processing system executes a data processing task, the data processing task is divided into multiple rounds of data processing tasks, and each round of data processing tasks includes: a data production task, and a data consumption task.

[0123] Figure 4 A flowchart of a data processing method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, this method is applied to Figure 3 In the data processing system shown, the method comprises the following steps:

[0124] S401. The time slice generation thread detects whether a first token representing that the data producer thread group has completed the production of the data production task corresponding to the Q-1 round and a second token representing that the data consumer thread group has completed the consumption of the data consumption task corresponding to the QP round are obtained.

[0125] Wherein, Q and P are both integers greater than or equal to 1.

[0126] When Q is greater than 1, in the data production task of the Q-1th round, when all the data producer threads in the data producer thread group complete the data production task of the Q-1th round, the first token of the Q-1th round is generated, and the token of the Q-1th round is sent to the time slice generation thread. When Q is equal to 1, at this time, the first token can be the initial first token.

[0127] When Q is greater than P, in the data processing task of the QPth round, when all the data consumer threads in the data consumer thread group complete the data processing task, the second token of the QPth round is generated. And the token of the QPth round is sent to the time slice generation thread. Among them, the value of P can be determined according to the number of data queues set by the data processing system. Among them, a data queue is used to store the data produced by a round of data producer thread groups. When Q is less than or equal to P, the task has not been executed before, at this time, the second token can be the initial second token.

[0128] The above-mentioned first initial first token and initial second token may be, for example, an initial first token and P initial second tokens generated when the management thread performs the initialization operation of the data processing task after receiving the data processing task.

[0129] S402: If yes, the time slice generation thread allocates the time slice corresponding to the Qth round of data production tasks to the data producer thread group based on the time point corresponding to the to-be-processed data and the time slice division parameters.

[0130] For example, when the wake-up condition is met, the time slice generation thread is awakened and generates the Qth round of time slices according to the task status of the current data to be processed.

[0131] The time slice division parameter is used to divide the time slice, and may include, for example: the number of time slices X. In this implementation, the length of a single time slice may be a fixed value, which may be a default value or a preset value corresponding to the data processing task.

[0132] Optionally, the time slice division parameters may also include: the number of time slices X, and the maximum time slice length Y. In this implementation, the size of the time slice between different rounds is adjustable, so that the size of the time slice can be dynamically adjusted based on the processing progress of the task, so that as many threads of the data production thread group as possible can execute the data production task in parallel, thereby improving the data generation efficiency.

[0133] For example, the time slice generation thread can determine the length of the time slice of the Qth round according to the relationship between the interval between the time point corresponding to the data to be processed and the current time and the first value, and the number of time slices X. The time slice generation thread allocates the time slice corresponding to the data production task of the Qth round to the data producer thread group according to the length of the time slice of the Qth round, the number of time slices X, and the time point corresponding to the data to be processed.

[0134] In the time slice corresponding to the data production task allocated to the Qth round, for example, the following steps may also be included:

[0135] (1) After completing the Q-1th round of data production tasks, the data producer thread group enters a blocked state until the Qth round of time slices are generated and the data producer thread group is awakened.

[0136] (2) The data producer thread group obtains the corresponding time slice and executes the Qth round of data production tasks based on the time slice.

[0137] The data processing method provided in the embodiment of the present application uses a token notification mechanism instead of a sleep waiting mechanism, and implements the advancement and scheduling of data tasks based on the token notification mechanism. The time slice generation thread generates a first token based on the data producer thread group completing the Q-1 round of data production tasks. After the data consumer thread group completes the data consumption task corresponding to the QP round, it generates a second token to allocate the time slice corresponding to the Q round of data production tasks. The data producer thread group starts the next round of data production tasks based on the time slice. This method effectively avoids invalid waiting caused by fixed sleep time in the sleep waiting mechanism. It reduces the idle time of the data producer thread group, improves the efficiency of data producer thread scheduling, and significantly enhances the overall data processing capability of the system.

[0138] The following example takes the case where, when the wake-up conditions are met, the time slice generation thread allocates the time slice corresponding to the Qth round of data production tasks to the data producer thread group based on the time point corresponding to the data to be processed and the time slice division parameters.

[0139] This example illustrates how to dynamically adjust the time slice based on the different states of the current data task by using the time slice to generate threads. Figure 5A flowchart of a method for allocating time slices corresponding to data production tasks by time slice generation threads provided in an embodiment of the present application is shown as follows: Figure 5 As shown, step S402 may also include, for example:

[0140] S501. The time slice generation thread determines the length of the time slice of the Qth round according to the relationship between the interval between the time point corresponding to the data to be processed and the current time and the first value, as well as the number of time slices.

[0141] The first value is a reference value between the time point corresponding to the data to be processed and the current time interval, and the first value is greater than or equal to the product of X and Y.

[0142] Exemplarily, when the interval between the time point and the current time is greater than a first value, the length of the time slice corresponding to the Qth round of data production tasks is equal to Y.

[0143] Among them, when the time interval is greater than the first value, it means that the current time is far away and the amount of unprocessed data accumulated by the data source is relatively large. Therefore, the batch data production state can be entered. On the basis of the fixed number of time slices X, the efficiency of concurrent data production by the data producer thread can be improved by increasing the time slice length. For example, the time slice length corresponding to the data production task of this round can be set to the maximum time slice length Y.

[0144] Exemplarily, when the interval between the time point and the current time is less than or equal to the first value, the length of the time slice corresponding to the data production task of the Qth round is equal to the quotient of Y and X.

[0145] Among them, when the time interval is less than or equal to the first value, it means that the current time is close, and the amount of unprocessed data accumulated by the data source is relatively small. Therefore, the real-time data production state can be entered. In this embodiment, the efficiency of concurrent data production by the data producer thread can be improved by reducing the time slice length. That is, a single time slice is further subdivided into multiple time slices so that each data producer thread can obtain a time slice to perform data production tasks. Thereby maintaining the concurrency of data production and the high efficiency of real-time task processing. Exemplarily, the time slice length of this round can be equal to the quotient of the maximum time slice length and the number of time slices X.

[0146] S502. The time slice generation thread obtains X time slices corresponding to the data production task of the Qth round based on the length of the time slice of the Qth round and the time point corresponding to the data to be produced.

[0147] S503: The time slice generation thread writes X time slices into a time slice queue.

[0148] Exemplarily, the time slice generation thread writes the generated X time slices into the time slice queue in chronological order.

[0149] By adding time slice queues to the time slice generation thread and the data producer thread group, the decoupling between the time slice generation thread and the data producer thread is achieved, and the synchronization mechanism is used to coordinate the access of the two to the time slice queue to ensure data consistency and thread safety, thereby achieving efficient data processing and resource utilization.

[0150] In the method provided by the embodiment of the present application, the time slice generation thread allocates the time slice corresponding to the data production task of this round to the data producer thread group based on the time point corresponding to the data to be processed and the time slice division parameters. By dynamically adjusting the time slice according to the different states of the current data task, when entering the batch data production state, the efficiency of the data producer thread concurrently producing data can be improved by increasing the time slice length. When entering the real-time data production state, the efficiency of the data producer thread concurrently producing data can be improved by reducing the time slice length. That is, a single time slice is further subdivided into multiple time slices, so that each data producer can obtain a time slice to perform data production tasks. Thereby maintaining the concurrency of data production and the high efficiency of real-time task processing. This method makes the data processing process more flexible, significantly improves the processing efficiency of data tasks, and greatly improves the overall processing speed of the data processing system.

[0151] Figure 6 A schematic diagram of a process for a data producer thread group to perform a data production task is provided for an embodiment of the present application, such as Figure 6 As shown, after the Qth round of time slice production, the following operations may also be included:

[0152] S601. The data producer thread in the data producer thread group enters the awakened state.

[0153] Exemplarily, after the data producer thread group completes the Q-1th round of data production tasks, the data producer threads in the data producer thread group enter a blocked state until the Qth round time slice is obtained, and the data producer threads in the data producer thread group are awakened.

[0154] S602: The data producer thread obtains the corresponding time slice from the time slice queue and executes the data production task of the time slice.

[0155] Exemplarily, the data producer threads in the data producer thread group concurrently obtain corresponding time slices from the time slice queue and execute the Qth data production task of the time slice.

[0156] S603: After the data producer thread produces data from the data source, it forms a data packet and stores the data packet in the data queue corresponding to the Qth round.

[0157] S604. The data producer thread adds 1 to the number of completed time slices corresponding to the Qth round to obtain an updated number of completed time slices.

[0158] Exemplarily, after the data producer thread in the data producer thread group obtains the time slice, it performs data production work. Each time the data producer thread completes a data production task, it correspondingly increases the number of completed time slices by 1, and obtains the updated number of completed time slices. The number of completed time slices is used to represent the total number of time slices completed by the current data producer thread group. That is, the parameter is a global variable of the data producer thread group.

[0159] S605. The data producer thread determines whether the Qth round of data production tasks is completed.

[0160] For example, if the updated number of completed time slices is equal to the number of time slices X, it means that the data producer thread is the last thread to complete the data generation task of this round of time slices. Therefore, the first token representing that the data producer thread group has completed the data production task corresponding to the Q round can be reported to the time slice generation thread.

[0161] Exemplarily, when the number of completed time slices is equal to the number of time slices X in the time slice parameter, it proves that the data production tasks of the Qth round have been completed. The last data producer thread in the data producer thread group to complete the Qth round of data production tasks generates a first token and reports the first token of the Qth round to the time slice generation thread.

[0162] Exemplarily, the last data producer thread in the data producer thread group to complete the Qth round of data production tasks can also generate a Qth round of data production completion flag and put it into the data queue corresponding to the Qth round. This flag can indicate to the data consumer thread group that the Qth round of data production tasks have been completed.

[0163] For another example, if the updated number of completed time slices is less than the number of time slices X, it indicates that the data production task of the Qth round is not completed, and then a new time slice is obtained from the time slice queue, and the data production task of the Qth round is continued.

[0164] Exemplarily, when the updated number of completed time slices is less than the number X of time slices in the time slice parameter, it proves that the data production task of the data producer thread group in the Qth round is not completed, and the data producer threads in the data producer thread group continue to obtain time slices from the time slice queue to perform data production tasks.

[0165] For another example, after the time slice generation thread allocates the time slice corresponding to the data production task of the Qth round, if the time slice generation thread has not obtained the first token indicating that the data producer thread group has completed the data production task corresponding to the Qth round when the first preset time is reached, a prompt message indicating that the data production of the Qth round is wrong is output.

[0166] The first preset duration refers to a time limit preset by the data processing system, which is used to control the time for the data producer thread group to complete the Qth round of data production tasks.

[0167] Exemplarily, the first preset duration may be pre-set by the management thread, or may be set by the time slice generation thread according to each round of data processing tasks. The embodiment of the present application does not limit the setting method of the first preset duration.

[0168] The prompt information may be, for example, error details recorded in a data processing system log, or a sound alarm or a flashing visual prompt may be issued on a system console or user interface. The embodiment of the present application does not limit the type of prompt information.

[0169] The method provided in the embodiment of the present application determines whether the data producer thread group has completed the data production task in the Q round by judging the relationship between the number of time slices completed by the data producer thread group in the Q round and the number X of time slices in the time slice parameter. When not completed, the data producer thread group enters a waiting state for wake-up. When the current work is completed, a first token representing the completion of the Q round of data tasks and a data production completion flag are generated. It is also determined by setting a first preset duration whether an error occurs in the data production task of the current round, and an error is promptly prompted after an error occurs. This method can ensure that when a problem occurs in data production, the current data processing task will not be erroneously advanced or skipped, and the system can quickly locate the problematic task for error handling, thereby avoiding data loss and ensuring the reliability of the data processing system and the integrity of the data.

[0170] Figure 7 A schematic diagram of a process for a data consumer thread group to perform a data consumption task is provided for an embodiment of the present application, such as Figure 7 As shown, after the data production thread writes the Q-th round data packet into the Q-th round data queue, the following steps may also be included:

[0171] S701. A data consumer thread in a data consumer thread group enters a wake-up state.

[0172] Exemplarily, after the data consumer thread group completes the Q-1th round of data processing, the data consumer threads in the data consumer thread group enter a blocked state until a data packet of the Qth round appears in the data queue, and the data consumer threads in the data consumer thread group are awakened.

[0173] S702: The data consumer thread obtains the corresponding data packet from the data queue corresponding to the Qth round, and executes the data consumption task of the data packet.

[0174] Exemplarily, the data consumer threads in the data consumer thread group respectively obtain the data packets in the Qth round data queue, and perform data processing tasks according to the content of the data packets. The embodiment of the present application does not limit the order in which the data consumer threads in the data consumer thread group obtain the data packets, and the data consumer threads obtain the data packets according to their own processing speed.

[0175] S703, the data consumer thread determines whether there is a new data packet in the Q-th round data queue, if so, execute S704, if not, execute S705.

[0176] S704: The data consumer thread continues to obtain new data packets from the data queue corresponding to the Qth round.

[0177] S705: Determine whether the current data consumer thread has received the Qth round of data production completion mark. If not, execute S706; if yes, execute S707.

[0178] S706: The Q-th round data producer thread has not completed the data production task, and the data consumer thread enters a blocked state, waiting for a new data packet.

[0179] S707: The data consumer thread group determines whether the number of data processing threads that have completed the current work is equal to N. If so, execute S708; if not, execute S709.

[0180] For example, when the data consumer thread group receives the Qth round of data production completion mark, the N data consumer threads are restarted respectively to determine whether the current data consumer thread has completed the data processing work. If the startup fails, it indicates that the current data consumer thread has not completed the data processing work. Whenever a data consumer thread is restarted, the number of completed threads is increased by 1 through atomic operations.

[0181] S708. The last data consumer thread in the data consumer thread group to complete startup reports a second token indicating that the data processing thread group has completed processing of the data processing task corresponding to the Qth round.

[0182] Exemplarily, when the number of data consumer threads is equal to N, the last data consumer thread in the data consumer thread group to complete the data processing task generates a second token representing the completion of data processing in this round, and reports the second token to the time slice generation thread to prove that the data processing thread group has completed the data processing task corresponding to this round.

[0183] S709: The data consumer thread enters a state of waiting for wake-up.

[0184] For example, if a data production completion flag is received and the number of data processing threads that have completed the current work is less than N, it is proved that the data production task has been completed. It is determined that an error has occurred in the data processing task of the data processing thread group in this round. The data processing thread group enters a waiting state for wake-up.

[0185] Exemplarily, after the time slice generation thread allocates the time slice corresponding to the data production task of the Qth round, if the time slice generation thread has not obtained the second token representing that the data consumer thread group has completed the data consumption task corresponding to the Qth round when the second preset time is reached, a prompt message representing that the data processing of the Qth round is wrong is output.

[0186] The second preset duration refers to a time limit preset by the data processing system, which is used to control the time for the data consumer thread group to complete the Qth round of data production tasks.

[0187] For example, the second preset duration may be pre-set by the management thread, or may be set by the time slice generation thread according to each round of data processing tasks. The embodiment of the present application does not limit the setting method of the first preset duration.

[0188] The method provided in the embodiment of the present application determines whether the data processing task of the data processing thread group in the Q round is completed by judging the relationship between the number of data consumer threads that have completed the Q round of data processing work and the total number of data consumer threads N. When the current work is not completed, the data consumer thread group enters a waiting state for wake-up. When the current work is completed, a second token representing the completion of the Q round of data processing tasks is generated. And by setting a second preset time length, it is determined whether an error occurs in the data consumption task of the current round, and an error is promptly prompted after an error occurs. This method can ensure that when a problem occurs in the data during the current round of processing, the current data processing task will not be erroneously advanced or skipped, and the system can quickly locate the task with the problem for error handling, thereby avoiding data loss and ensuring the reliability of the data processing system and the integrity of the data.

[0189] The following is a specific example to illustrate how to implement data processing for time slice round advancement. Figure 8 A flowchart of implementing data processing based on a data processing system is provided in an embodiment of the present application. Fig. 9 A flow chart of a data processing method for implementing time slice round advancement provided in an embodiment of the present application, taking the current data processing task round as the Qth round as an example, Fig. 9 As shown, the method may include the following steps:

[0190] S901, after completing the data production task of the Q-1th round, the data producer thread in the data producer thread group generates the first token of the Q-1th round and sends the first token to the time slice generation thread. The data producer thread group enters a blocked state.

[0191] S902: The time slice generation thread obtains the first token of the Q-1th round and the second token of the QPth round, and determines that the time slice generation thread wake-up condition is met.

[0192] Among them, when the number of P is greater than or equal to 2, it can ensure that the Q-th round data producer thread group can perform the Q-th round data production task without waiting for the Q-1-th round data consumer task to be completed. The Q-th round data production task can be advanced to avoid the situation where the data consumer thread waits ineffectively due to the long time-consuming data production task.

[0193] S903. The time slice generation thread allocates the Q-th round time slice and puts the Q-th round time slice into the time slice queue.

[0194] like Figure 8 As shown in a, the time slice generation module puts the allocated time slice into the time slice queue.

[0195] S904: The data producer thread in the data producer thread group is awakened, concurrently obtains the time slice in the time slice queue, and performs data production tasks based on the time slice.

[0196] like Figure 8 As shown in b and c, the data producer thread obtains the time slice in the time slice queue and performs the data production task.

[0197] S905. The data producer thread group puts the produced data packets into the data queue corresponding to the Qth round.

[0198] like Figure 8 As shown in (d), the data producer thread group puts the produced data packets into the corresponding data queue.

[0199] S906: The data consumer threads in the data consumer thread group are awakened, concurrently obtain data packets in the data queue, and perform data processing according to the content of the data packets.

[0200] like Figure 8As shown in e, the data consumer thread obtains the data packet and processes the data.

[0201] S907. After the task of the data producer thread group in the Qth round is completed, the last data producer thread in the data producer thread group to complete the task generates the first token of the Qth round and sends the first token of the Qth round to the time slice generation thread.

[0202] like Figure 8 As shown in f, the data producer thread sends the first token to the time slice generation thread.

[0203] S908. After completing the Qth round of data consumption tasks, the data consumer thread group generates a second token for the Qth round and sends the second token to the time slice generation thread.

[0204] like Figure 8 As shown in g, the data consumer thread sends the second token to the time slice generation thread.

[0205] S909. After obtaining the first token of the Qth round and the second token of the Q-P+1th round, the time slice generation thread ensures that all data processing of the Qth round is completed.

[0206] In the method provided by the embodiment of the present application, the time slice generation thread allocates the time slice corresponding to the data production task of this round to the data producer thread group based on the time point corresponding to the data to be processed and the time slice division parameters. By dynamically adjusting the time slice according to the different states of the current data task, when entering the batch data production state, the efficiency of the data producer thread concurrently producing data can be improved by increasing the time slice length. When entering the real-time data production state, the efficiency of the data producer thread concurrently producing data can be improved by reducing the time slice length. That is, a single time slice is further subdivided into multiple time slices, so that each data producer can obtain a time slice to perform data production tasks. Thereby maintaining the concurrency of data production and the high efficiency of real-time task processing. This method makes the data processing process more flexible, significantly improves the processing efficiency of data tasks, and greatly improves the overall processing speed of the data processing system.

[0207] The received first token and second token are used to determine whether the data processing task of the current round is completed. If it is not completed or an error occurs, corresponding processing is performed. This method can ensure that when a problem occurs in data production, the current data processing task will not be mistakenly advanced or skipped. The system can quickly locate the task with the problem and handle the error, thereby avoiding data loss and ensuring the reliability of the data processing system and the integrity of the data.

[0208] Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10 As shown, the electronic device 1000 may include: a memory 1001 and a processor 1002. The electronic device may be, for example, the aforementioned BMS controller, i.e., BMC. Optionally, the electronic device may further include a transceiver 1003, wherein the memory 1001 and the processor 1002 communicate; illustratively, the memory 1001, the processor 1002 and the transceiver 1003 may communicate via a communication bus 1004, the memory 1001 is used to store a computer program, and the processor 1002 executes the computer program to implement the method of the above embodiment. The transceiver may be, for example, a CAN transceiver.

[0209] Optionally, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps in the method embodiments disclosed in the present application may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0210] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, a method in any of the above method embodiments is implemented.

[0211] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in any of the above method embodiments when the computer program is executed by a processor.

[0212] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The above-mentioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the above-mentioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape, floppy disk, optical disc and any combination thereof.

[0213] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0214] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0216] Continue to refer to Figure 3 The embodiment of the present application also provides a data processing system, which runs a time slice generation thread, a data producer thread group, and a data consumer thread group. Optionally, a management thread can also be run, and a data queue and a time slice queue are set.

[0217] The data producer thread in the data producer thread group is used to execute the data production task described in the above method embodiment.

[0218] The data consumer threads in the data consumer thread group are used to execute the data consumption tasks described in the above method embodiment on the data produced by the data producer thread group.

[0219] The time slice generation thread is used to execute the task of the time slice generation thread in the above method embodiment.

[0220] The management thread is used to execute the task of managing the thread in the above method embodiment.

[0221] The time slice queue is used to store the content of the time slice queue in the above method embodiment.

[0222] The data queue is used to store the content of the data queue in the above method embodiment.

[0223] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0224] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: The method is applied to a data processing system, wherein the data processing system runs a time slice generation thread, a data producer thread group, and a data consumer thread group; the method comprises: The time slice generation thread detects whether a first token representing that the data producer thread group has completed the production of the data production task corresponding to the Q-1th round, and a second token representing that the data consumer thread group has completed the consumption of the data consumption task corresponding to the QP round, are obtained; the Q and the P are both integers greater than or equal to 1; If so, the time slice generation thread allocates the time slice corresponding to the Qth round of data production tasks to the data producer thread group based on the time point corresponding to the to-be-processed data and the time slice division parameters.

2. The method according to claim 1, characterized in that The time slice division parameters include: the number of time slices X, and the maximum time slice length Y; the time slice generation thread allocates the time slice corresponding to the data production task of the Qth round to the data producer thread group based on the time point corresponding to the data to be processed and the time slice division parameters, including: The time slice generation thread determines the length of the time slice of the Qth round according to the relationship between the interval between the time point and the current time and the first value, and X; the first value is greater than or equal to the product of X and Y; The time slice generation thread obtains X time slices corresponding to the data production task of the Qth round based on the length of the time slice of the Qth round and the time point corresponding to the data to be processed; The time slice generation thread writes X time slices into the time slice queue.

3. The method according to claim 2, characterized in that When the interval between the time point and the current time is greater than the first value, the length of the time slice corresponding to the Qth round of data production tasks is equal to the Y.

4. The method according to claim 2, characterized in that: When the interval between the time point and the current time is less than or equal to the first value, the length of the time slice corresponding to the data production task of the Qth round is equal to the quotient of Y and X.

5. The method according to claim 2, characterized in that: The method further comprises: The data producer threads in the data producer thread group perform the following operations: Enter the awakening state; Obtaining a corresponding time slice from the time slice queue and executing a data production task for the time slice; Add 1 to the number of completed time slices corresponding to the Qth round to obtain an updated number of completed time slices; If the updated number of completed time slices is equal to X, reporting to the time slice generation thread a first token indicating that the data producer thread group has completed the data production task corresponding to the Qth round; If the updated number of completed time slices is less than X, new time slices continue to be obtained from the time slice queue.

6. The method according to claim 5, characterized in that The data processing system is provided with P data queues, each data queue is used to store data packets generated by a round of data consumption tasks; The data consumption task of executing the time slice includes: Produce data of the time slice from the data source to obtain the data packet of the time slice; The data packets of this time slice are stored in the data queue corresponding to the Qth round.

7. The method according to claim 1, characterized in that The data consumer threads in the data consumer thread group perform the following operations: Enter the awakening state; Obtain a corresponding data packet from the data queue corresponding to the Qth round, and execute a data consumption task for the data packet; If there are new data packets in the data queue, continue to obtain new data packets from the data queue corresponding to the Qth round; If there is no new data packet in the data queue, and the number of data consumer threads that have completed the current work is equal to N, then a second token representing that the data consumer thread group has completed the consumption of the data consumption task corresponding to the Qth round is reported to the time slice generation thread; If there is no new data packet in the data queue and the number of data consumer threads that have completed the current work is less than N, it enters the waiting wake-up state.

8. The method according to claim 1, characterized in that: The method further comprises: After allocating the time slice corresponding to the data production task of the Qth round, if the time slice generation thread has not obtained the first token indicating that the data producer thread group has completed the data production task corresponding to the Qth round when the first preset time is reached, a prompt message indicating that the data production of the Qth round is wrong is output.

9. The method according to claim 1, characterized in that: The method further comprises: After allocating the time slice corresponding to the data production task of the Qth round, if the time slice generation thread has not obtained the second token representing that the data consumer thread group has completed the data consumption task corresponding to the Qth round when reaching the second preset time length, a prompt message representing that the data processing of the Qth round has an error is output.

10. The method according to claim 5, characterized in that The data processing system also runs a management thread, and the method further includes: The management thread performs an initialization operation based on the data processing task to set the value of one or more of M, N, X, Y, and P.

11. The method according to claim 10, characterized in that The method further comprises: The management thread generates an initial first token and an initial second token; the initial first token and the second token are used to trigger the time slice generation thread to allocate the time slice corresponding to the first round of data production tasks; The management thread sends the initial first token and the initial second token to the time slice generation thread.

12. An electronic device, characterized in that: The electronic device comprises: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 11 when executed by a processor.

14. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 11 when being executed by a processor.

15. A data processing system, characterized in that: The data processing system runs a time slice generation thread, a data producer thread group, and a data consumer thread group; The data producer thread in the data producer thread group is used to execute data production tasks; The data consumer threads in the data consumer thread group are used to execute data consumption tasks on the data produced by the data producer thread group; The time slice generation thread is used to: Detect whether a first token representing that the data producer thread group has completed the production of the data production task corresponding to the Q-1 round and a second token representing that the data consumer thread group has completed the consumption of the data consumption task corresponding to the QP round are obtained; Both Q and P are integers greater than or equal to 1; If so, the time slice generation thread allocates the time slice corresponding to the Qth round of data production tasks to the data producer thread group based on the time point corresponding to the to-be-processed data and the time slice division parameters.

Images