Data processing method and apparatus, and electronic device and medium

Abstract

A data processing method, which belongs to the technical field of cloud computing. The method comprises: within any time slice allocated to a consumer driver, when the total number of bytes recorded by a memory counter drops to 0 or data in a buffer area is read out, removing a producer driver from a blocking queue and then adding the producer driver to a ready queue, and controlling the consumer driver to be in a waiting state without yielding the allocated time slice; after the waiting duration of the consumer driver reaches a preset duration, allocating a time slice to the producer driver in the ready queue, and waking up the consumer driver by means of the producer driver; and controlling the producer driver to write data into the buffer area, and controlling the consumer driver to read data from the buffer area. Further disclosed are a data processing apparatus, and an electronic device and a medium.

Description

Data processing method and device, electronic equipment and medium

[0001] The present disclosure claims priority to Chinese Patent Application No. 202411237701.7, filed on September 4, 2024, with the Chinese Patent Office, entitled "Data processing method and device, electronic equipment and medium", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of cloud computing, and particularly relates to a data processing method and device, electronic equipment and medium. BACKGROUND

[0003] In a multi-task operating environment, a system (including an operating system or a database system) needs to manage multiple concurrent query requests and transaction processing. To adapt to the multi-task operating requirements, the system can adopt a time-slice scheduling strategy and a Pipeline execution framework (also referred to as an execution pipeline or a query pipeline). The time-slice scheduling strategy is a process scheduling strategy based on time allocation, which is a method used by a scheduler of the system to allocate processor time to concurrent tasks. The Pipeline execution framework refers to a strategy of organizing and executing components or operations of an execution plan in a pipeline form in system query processing. For the system adopting the time-slice scheduling strategy and the Pipeline execution framework, the speed of writing data by a producer driver upstream of a time-slice task is too fast, and the speed of reading data by a consumer driver is too slow, which leads to system memory shortage, overload or crash, etc.

[0004] However, for the system adopting the time-slice scheduling strategy and the Pipeline execution framework, how to perform data processing becomes a problem to be solved at present. SUMMARY

[0005] Embodiments of the present disclosure provide a data processing method and device, electronic equipment and medium for a system adopting a time-slice scheduling strategy and a Pipeline execution framework. The technical solution is as follows:

[0006] In a first aspect, a data processing method is provided, and the method is applied to a system adopting a time-slice scheduling strategy and a Pipeline execution framework, and the method comprises:

[0007] the memory counter records the total number of bytes of data currently stored in the cache area is reduced to 0 or the data in the cache area is read empty, the producer driver is taken out from the blocking queue and added to the ready queue, the blocking queue is used to store the producer driver which stops writing data due to being in the blocking state by giving up the allocated time slice, the ready queue is used to store the producer driver which is in the ready state and waits for the time slice to be allocated, and the consumer driver is controlled to be in the waiting state without giving up the allocated time slice;

[0008] after the waiting duration of the consumer driver reaches a preset duration, the time slice is allocated to the producer driver in the ready queue, so that the producer driver is in the running state, and the consumer driver is woken up by the producer driver, so that the consumer driver is in the running state;

[0009] the producer driver in the running state is controlled to write data into the cache area, and the consumer driver in the running state is controlled to read data from the cache area.

[0010] In a second aspect, a data processing device is provided, which is applied to a system adopting a time slice scheduling strategy and a Pipeline execution framework, and the device comprises:

[0011] a first adding module, configured to, in any time slice allocated to a consumer driver, when the total number of bytes of data currently stored in a cache area recorded by a memory counter is reduced to 0 or the data in the cache area is read empty, take a producer driver out from a blocking queue and add it to a ready queue, the blocking queue is used to store the producer driver which stops writing data due to being in the blocking state by giving up the allocated time slice, the ready queue is used to store the producer driver or consumer driver which is in the ready state and waits for the time slice to be allocated;

[0012] a first control module, configured to control the consumer driver to be in the waiting state without giving up the allocated time slice,

[0013] an allocating module, configured to, after the waiting duration of the consumer driver reaches a preset duration, allocate the time slice to the producer driver in the ready queue, so that the producer driver is in the running state;

[0014] a waking-up module, configured to wake up the consumer driver by the producer driver, so that the consumer driver is in the running state;

[0015] a second control module, configured to control the producer driver in the running state to write data into the cache area.

[0016] The third control module is configured to control the consumer driver in the running state to read data from the cache area.

[0017] In a third aspect, an electronic device is provided, including a processor and a memory; the memory stores at least one program code; the at least one program code is used to be called and executed by the processor to implement the data processing method in the first aspect.

[0018] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores at least one computer program; the at least one computer program is executed by a processor to implement the data processing method in the first aspect.

[0019] In a fifth aspect, a computer program product is provided, and the computer program product includes a computer program; the computer program is executed by a processor to implement the data processing method in the first aspect.

[0020] The technical scheme provided by the embodiments of the present disclosure has the following beneficial effects:

[0021] For a system adopting a time slice scheduling strategy and a Pipeline execution framework, the producer driver and the consumer driver are both allocated time slices, the data writing operation of the producer driver and the data reading operation of the consumer driver are executed in parallel within the allocated time slices, and the speed of the producer driver in writing data into the cache area is faster than the speed of the consumer driver in reading data from the cache area. The storage space of the cache area is limited. In order to monitor the storage space of the cache area, the disclosure embodiment sets a memory counter for recording the total number of bytes of data currently stored in the cache area. When the total number of bytes of data currently stored in the cache area recorded by the memory counter reaches a byte number threshold, the producer driver is added to a blocking queue, so that the producer driver gives up the allocated time slice and stops writing data into the cache area, avoiding the continuous increase of the memory of the cache area. When the total number of bytes of data currently stored in the cache area recorded by the memory counter decreases to 0 or the data in the cache area is read empty, data needs to be written into the cache area. At this time, the producer driver can be taken out of the blocking queue and added to a ready queue, and the consumer driver is controlled to be in a waiting state without giving up the allocated time slice. After the waiting time of the consumer driver reaches a preset time, the producer driver in the ready queue is allocated a time slice, so that the producer driver is in a running state, and the consumer driver in the waiting state is awakened by the producer driver in the running state. Then the producer driver in the running state is controlled to write data into the cache area, and the consumer driver in the running state is controlled to read data from the cache area, so as to realize the reading and writing of data in the cache area. Considering that the speed of the producer driver in writing data into the cache area is faster than the speed of the consumer driver in reading data from the cache area, when there is more data stored in the cache area, the waiting time of the producer driver is longer. By controlling the producer driver to give up the time slice, the system resources are avoided from being occupied by the producer driver in vain, and the system performance is improved. However, the scheduling of the driver needs to consume system resources, and when there is less data stored in the cache area, the waiting time of the consumer driver is shorter. By controlling the consumer driver to wait in place without giving up the allocated time slice, the consumption of system scheduling resources is reduced, and the system performance is improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the disclosure, and other drawings can also be obtained by those skilled in the art without any creative effort based on these drawings.

[0023] FIG. 1 is a schematic diagram of a Pipeline execution framework;

[0024] FIG. 2 is a schematic diagram of a time slice scheduling process;

[0025] FIG. 3 is an architecture diagram of a system according to an embodiment of the present disclosure;

[0026] FIG. 4 is a flowchart of a data processing method according to an embodiment of the present disclosure;

[0027] FIG. 5 is a timing diagram of a back pressure process according to an embodiment of the present disclosure;

[0028] FIG. 6 is a graph of a trend of a memory water level in a buffer area in a standard time slice over time according to an embodiment of the present disclosure;

[0029] FIG. 7 is a schematic diagram of a calculation process of a first speed of a producer driver writing data and a second speed of a consumer driver reading data according to an embodiment of the present disclosure;

[0030] FIG. 8 is a schematic diagram of a structure of a data processing apparatus according to an embodiment of the present disclosure;

[0031] FIG. 9 shows a structural block diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] For the purposes of the present disclosure, the technical solutions and advantages will be more clearly apparent from the following detailed description of the embodiments of the present disclosure, taken in conjunction with the accompanying drawings.

[0033] It can be understood that the terms "each", "multiple", "any" and the like used in the embodiments of the present disclosure include two or more, each refers to each of the corresponding multiple, and any refers to any one of the corresponding multiple. For example, multiple words include 10 words, and each word refers to each of the 10 words, and any word refers to any one of the 10 words.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0035] With the development of the Internet, real-time data stream, and the diversification of connected devices, and the demand for search services, social networks, mobile commerce, and open collaboration, cloud computing has rapidly developed. Unlike previous parallel distributed computing, the emergence of cloud computing will revolutionize the entire Internet model and enterprise management model. Cloud computing refers to the delivery and use of IT infrastructure as a service. It refers to obtaining the required resources in a scalable manner on demand through the network. Broadly, cloud computing refers to the delivery and use of services. It refers to obtaining the required services in a scalable manner on demand through the network. Such services can be IT and software, Internet related, or other services. Cloud computing is the product of the development of grid computing, distributed computing, parallel computing, utility computing, network storage, virtualization, load balancing, and other traditional computer and network technologies.

[0036] The embodiments of the present disclosure relate to the database direction in cloud computing technology. In short, a database can be regarded as an electronic file cabinet - a place for storing electronic files. Users can perform operations such as adding, querying, updating, and deleting data in the files. A "database" is a collection of data stored together in a certain way, shared by multiple users, with as little redundancy as possible, and independent of application programs. A database management system (DBMS) is a computer software system designed to manage databases. It generally has basic functions such as storage, interception, security, backup, etc. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language they use, such as SQL (Structured Query Language), XQuery; or according to the performance focus, such as maximum size, highest speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, such as supporting multiple query languages at the same time.

[0037] Before implementing the embodiments of the present disclosure, the terms related to the embodiments of the present disclosure are first explained.

[0038] Pipeline execution framework

[0039] In a database system, the execution of a query involves many steps, including multiple operators (such as join operation, selection operation, projection operation, etc.), each of which can need to process a large amount of data. These operators can be executed in series, i.e., one operation is completed before the next operation starts. However, in this way, some operations can have to wait for the previous operation to complete before they can start, which can result in low computational efficiency. To improve computational efficiency, the Pipeline execution framework is proposed, which is inspired by the instruction pipelining technique in computer science. The Pipeline execution framework, also known as execution pipeline or query pipeline, refers to a strategy of organizing and executing the components or operations of an execution plan in a pipelined manner in database query processing. With the Pipeline execution framework, one operation can start processing the output of the previous operation while the current operation is processing the current batch of data. Since the Pipeline execution framework allows different query stages to be processed simultaneously on multiple cores or processors, it greatly improves the utilization of resources and the parallelism of the entire query processing. Therefore, most database systems currently use the Pipeline execution framework when executing queries to optimize their performance.

[0040] The Pipeline execution framework includes two modes of operator execution, which are:

[0041] Blocking execution: an operation needs to wait for all its inputs to be processed before it can be executed.

[0042] Non-blocking execution: an operator can immediately process the output of the upstream operator without waiting for all inputs to be processed.

[0043] In the Pipeline execution framework, each Pipeline is composed of a set of operators. The same Pipeline, if it can be executed in parallel, forms a Driver (driver program) on each parallel degree. The Driver is the smallest unit of execution and scheduling in the Pipeline framework. A Chunk is a unit of data, and the Driver in the Pipeline can write to the Chunk or read the Chunk. The Driver in the Pipeline that writes to the Chunk is usually referred to as a producer Driver, and the Driver in the Pipeline that reads the Chunk is usually referred to as a consumer Driver. For example, the Pipeline execution framework shown in FIG. 1 includes two Pipelines, Pipeline 1 and Pipeline 2. Pipeline 1 includes Driver 1 and Driver 2, and Driver 1 and Driver 2 include a network reader (NetWork Reader) and a hash aggregation consumer (HashAgg Consume), respectively. Pipeline 2 includes Driver 1 and Driver 2, and Driver 1 and Driver 2 include a hash aggregation producer (HashAgg Producer) and an output (Ouput), respectively. Driver 1 and Driver 2 in Pipeline 1 can generate Chunks and output the generated Chunks to Driver 1 and Driver 2 in Pipeline 2. The two Drivers in Pipeline 1 are referred to as producer Drivers, and the two Drivers in Pipeline 2 are referred to as consumer Drivers.

[0044] Since there is often a large difference between the speed at which the producer Drivers generate Chunks and the speed at which the consumer Drivers consume Chunks, a back pressure mechanism needs to be introduced to avoid problems such as insufficient memory and system overload.

[0045] Time-slice scheduling

[0046] Time slice scheduling is a process scheduling strategy based on time allocation. In a multitasking operating environment, database systems need to handle multiple concurrent query requests and transactions. Time slice scheduling is one of the methods used by the operating system's scheduler to allocate processor time to these concurrent tasks. In the time slice scheduling mechanism, the operating system or database management system scheduler allocates a time period, called a time slice, to each running task (process or thread). Each task is allowed to execute within its allocated time slice. Once a task's time slice is up, even if the task has not completed, the scheduler deprives the task of control over the processor and places it at the end of the ready queue, then schedules another task to run its time slice.

[0047] The use of time slice scheduling has the following advantages:

[0048] Relative fairness of tasks: Each task has the opportunity to obtain execution time, preventing any single task from monopolizing the processor.

[0049] Good responsiveness: By subdividing the time slice, the system can respond quickly to interactive tasks.

[0050] Wide applicability: Time slice scheduling is widely used in various types of operating systems, including real-time systems and general-purpose systems.

[0051] Generally, each Driver in the time slice scheduling process will transition between the ready-run-block three states. Referring to Figure 2, after the Driver is created, it is first in the ready state and is written into the ready queue. The Drivers in the ready queue have priorities, and high-priority Drivers have priority in obtaining time slices. After the Driver in the ready queue obtains a time slice, it is in the running state. During the running process, the Driver in the running state will have two situations:

[0052] 1) After a time slice length period, the normal execution ends, the time slice is given up, and the Driver enters the ready queue;

[0053] 2) It meets a certain condition and is in the blocked state, and joins the blocked queue. When it meets another condition, it enters the ready queue and is in the ready state.

[0054] Backpressure mechanism

[0055] Backpressure is a common concept in flow control and system design, especially in reactive programming and network communication. It is a strategy or mechanism to prevent fast producers from overwhelming slow consumers, leading to overload, data loss, or system crashes. In simple terms, backpressure is a feedback control mechanism that allows consumers to signal producers to slow down or pause sending more data until the consumer has processed existing data and is ready to receive new data. Backpressure helps maintain system stability and efficiency when producers and consumers do not match in speed.

[0056] There are many ways to implement backpressure, including but not limited to:

[0057] Pause and resume: Consumers can tell producers to pause sending data, and when the consumer is ready to receive new data, resume sending.

[0058] Buffer management: Use a buffer to store data sent by producers, and when the buffer capacity reaches a threshold, notify the producer to slow down or pause sending; conversely, when the buffer data is consumed by a certain amount, notify the producer to continue sending data.

[0059] Request-response mode: Consumers actively request a certain amount of data from producers based on their processing capacity, and producers send the corresponding amount of data based on the request.

[0060] In the field of reactive programming, such as systems using the Reactive Streams specification (such as RxJava, Reactor, Akka, Streams, etc.), backpressure is a core function that implements the above flow control behavior in a standardized way, allowing reactive systems to handle asynchronous data streams efficiently and robustly. Common backpressure mechanisms in the industry include Flink backpressure, Spark backpressure, and backpressure in Java blocking queues, which will be introduced below.

[0061] Flink backpressure

[0062] Apache Flink is a distributed stream processing framework for processing large-scale data streams. In a distributed stream processing system, various computing nodes (tasks, operators, subtasks) process and transmit data in a pipeline manner. When the downstream operations cannot process the data sent by the upstream operations in a timely manner, backpressure occurs. Flink's backpressure mechanism acts on its task scheduling system. When the downstream task processing speed is lower than the upstream task data generation speed, the backpressure mechanism will notify the upstream task to reduce the sending speed, thereby avoiding memory overflow and performance degradation of the entire system. Flink ensures the stability and reliability of stream processing through this way.

[0063] Flink's backpressure is triggered by downstream tasks, and the backpressure process is controlled and managed in the following ways:

[0064] Buffer filling: Flink's data transmission is through network buffers. When an operation's output buffer is filled, data will be sent to the downstream operation.

[0065] Signal downstream task: If the downstream task is processing data, and a new data packet arrives, and the downstream task does not have enough empty data buffer space to receive it, the upstream task will be notified to stop sending data.

[0066] Upstream adjustment: When the upstream task receives the backpressure signal, it will stop sending data and store it in its own buffer. During this process, the upstream task will enter a waiting state until the downstream task has processed the existing data and is ready to receive new data.

[0067] Release backpressure: When the downstream task's buffer has available space, it will notify the upstream task, and then the upstream task can continue to send data.

[0068] Monitoring and diagnosis: Flink provides monitoring tools that allow users to view whether tasks are experiencing backpressure. For example, through Flink's Web UI, users can see which tasks are experiencing backpressure, and their status is marked as "backpressure".

[0069] Using the backpressure mechanism, Flink can automatically adjust the processing speed between tasks to ensure that all tasks work together and avoid data loss or system crashes due to speed mismatch. At the same time, the backpressure mechanism is also an important indicator for evaluating and optimizing the performance of stream processing jobs. If a system frequently encounters backpressure, it may indicate a performance bottleneck that needs to be addressed by optimizing task parallelism, adjusting buffer size, or improving task computing logic.

[0070] Spark backpressure mechanism

[0071] In systems without effective backpressure mechanisms, if the data generation rate consistently exceeds the processing rate, the system's memory, cache, and other resources will be quickly depleted, and processing delays will significantly increase, eventually leading to system collapse. In Apache Spark, to prevent the problem of data production rate exceeding Spark processing rate and causing system resource depletion, the Spark backpressure mechanism is adopted. The Spark backpressure mechanism is mainly for Spark Streaming applications, which is one of the components of Spark and is used to process real-time data streams. The principle of Spark backpressure mechanism is as follows:

[0072] Rate Limiting: Spark Streaming introduces a Dynamic Rate Limiter, which can dynamically adjust the speed of incoming data based on the processing system's capabilities. When starting a job, Spark Streaming can set a maximum receiving rate (spark.streaming.receiver.maxRate) and a maximum rate per partition (spark.streaming.kafka.maxRatePerPartition) to control the input flow rate of data.

[0073] Monitoring Processing Delay: Spark Streaming monitors the processing delay during processing. If the system detects that the processing time exceeds the interval time of batch calculation, and the consumption speed is not enough to keep up with the data generation speed, the Spark backpressure mechanism will be started.

[0074] Adjusting Rate: Based on the processing time and delay of several historical batches, Spark Streaming automatically adjusts the data receiving rate by increasing or decreasing the rate to adapt to the current processing capacity. This dynamic adjustment helps to prevent system overload and maintain processing stability.

[0075] Feedback Control: This mechanism is essentially a feedback control system that automatically adjusts the input rate based on real-time performance feedback (such as processing delay) of Spark Streaming jobs, thereby avoiding overload.

[0076] Currently, in Spark 2.x and higher versions, the Spark backpressure mechanism is enabled by default, and it can be explicitly enabled by setting spark.streaming.backpressure.enabled=true.

[0077] Backpressure mechanism in Java blocking queue

[0078] The BlockingQueue class in Java actually contains built-in backpressure mechanisms to control the flow of data between producers and consumers, preventing resource exhaustion and providing flow control. Java mainly uses the following two mechanisms to achieve backpressure:

[0079] 1. Capacity Limitation: BlockingQueue can specify a capacity when created, which means the queue can only hold a limited number of elements. When the queue is full, threads trying to add elements to it will be blocked until there is space available. Since producers have to wait for consumers to take elements from the queue before adding elements to the queue, this is a form of backpressure.

[0080] 2. Blocking Operations: BlockingQueue provides several methods such as put and take, which block threads when the queue state does not meet the operation requirements. For example, the put method blocks the thread when the queue is full until there is space available, and the take method blocks the thread when the queue is empty until there are elements available. These blocking properties support natural backpressure on producers when consumers are not enough to handle all the produced data.

[0081] In the underlying implementation, BlockingQueue may use different synchronization or concurrency mechanisms such as ReentrantLock or Semaphore depending on the implementation class (such as ArrayBlockingQueue, LinkedBlockingQueue, PriorityBlockingQueue, SynchronousQueue, etc.).

[0082] ReentrantLock: Used for mutual exclusion access to queue elements, in implementations such as ArrayBlockingQueue, the lock is used to protect the state of the queue and supports condition variables such as not Full and not Empty to implement blocking behavior.

[0083] Semaphore: Semaphore can be used to control the number of concurrent accesses to the queue, although the Java BlockingQueue interface implementation usually does not directly use Semaphore, but some implementations can use the semaphore pattern to control concurrent access to resources.

[0084] ArrayBlockingQueue internally uses a lock (ReentrantLock) and two condition variables (usually of type Condition). A producer, when trying to put an element, will wait on the notFull condition if the queue is full. A consumer, after taking an element from the queue, will wake up possibly waiting producers by notFull.signal() if the queue is not full anymore. Vice versa, a consumer, when trying to take an element, will wait on the notEmpty condition if the queue is empty. A producer, after putting an element into the queue, will wake up possibly waiting consumers by notEmpty.signal() if the queue is not empty.

[0085] In summary, the BlockingQueue in Java has a back pressure design, and by using the synchronization mechanism such as mutual exclusion lock and condition variable (possibly similar to the concept of semaphore) in its implementation, it provides flow control function between producers and consumers.

[0086] In summary, the above several back pressure mechanisms are suitable for scenarios related to network flow control, and do not use time slice scheduling strategy, and do not need to consider the impact of scheduling itself, and cannot be applied to systems that use time slice scheduling strategy and Pipeline execution framework. For systems that use time slice scheduling strategy and Pipeline execution framework, in general, the following problems need to be addressed:

[0087] (1) The data read and write speed of the producer Driver and the consumer Driver is quite different, and the speed of the producer Driver writing data into the cache area is much greater than the speed of the consumer Driver reading data from the cache area.

[0088] (2) The data read and write speed of the producer Driver and the consumer Driver can be measured, but cannot be intervened.

[0089] (3) The scheduling overhead is large when the context of the producer Driver and the consumer Driver is switched. If the context is frequently switched within a time slice, it will lead to the decline of system performance.

[0090] (4) The size of the cache area has an upper limit, and the memory space is insufficient.

[0091] In view of the problems of the system using the time slice scheduling strategy and the Pipeline execution framework, in order to be able to process data based on the back pressure mechanism in the system using the time slice scheduling strategy and the Pipeline execution framework, the embodiment of the disclosure provides a system using the time slice scheduling strategy and the Pipeline execution framework, referring to FIG. 3, the system includes a producer Pipeline, a cache area and a consumer Pipeline.

[0092] The parallelism of the producer Pipeline is m, and the producer Pipeline includes m producer Drivers. Each producer Driver includes multiple operators, and the last operator can be a LocalExchange operator.

[0093] The parallelism of the consumer Pipeline is n, and the consumer Pipeline includes n consumer Drivers. Each consumer Driver has a cache area (LocalBuffer).

[0094] The number of the cache areas is the same as the number of the consumer Drivers, and can be n. Each cache area provides a read-write interface, and can receive data (Chunks) sent by the LocalExchange operator of the upstream producer Driver to the downstream. Each cache area includes a storage queue, and the storage queue can be an unbounded non-blocking queue and belongs to a linked list structure. When the LocalExchange operator of the producer Driver generates a Chunk, the Chunk is added to the head of the linked list of a cache area. When the consumer Driver needs to read the Chunk from the cache area, the Chunk is read from the tail of the linked list of the cache area, and the read Chunk is removed from the tail of the linked list. After the consumer Driver operates the Chunk, the memory occupied by the Chunk is recycled or destroyed.

[0095] For the system using the time slice scheduling strategy and the Pipeline execution framework, under the condition of not limiting the memory usage and assuming that the memory of the database is large enough, the m producer Drivers and the n consumer Drivers can freely read and write the n cache areas, the m producer Drivers and the n consumer Drivers do not interfere with each other, and after using the respective time slices, that is, after running for a time slice length, the allocated time slices are given up, and then the producer Drivers or the consumer Drivers are added to the pending queue. After the new time slices are allocated, the waiting thread group can take the producer Drivers or the consumer Drivers from the pending queue, and then perform the execution of the next round of time slices.

[0096] However, in actual application scenarios, the use of memory is limited, the memory of the database is limited, and the speed at which the producer Driver generates and writes the Chunk is much greater than the speed at which the consumer Driver reads and destroys the Chunk, which causes the memory occupancy of the database to continuously and rapidly increase until an out of memory phenomenon occurs.

[0097] To solve this problem, the system provided by the embodiments of the present disclosure further includes a memory counter (memory_ledger) shared by the n buffers, used to record the total number of bytes of data currently stored by the n buffers. By setting the memory counter, the memory water level can be determined based on the total number of bytes of data currently stored by the buffer recorded by the memory counter, so that the data read and write operations of the system can be controlled. Since the total number of bytes is recorded, the overhead of serialization, deserialization, and memory copying of the buffer can be reduced, and the system performance is improved.

[0098] Further, based on the understanding of the SettableFuture class in Java, two state quantities, namely a writable signal (i.e., a first signal) and a readable signal (i.e., a second signal), are introduced. The writable signal and the readable signal are an instance of the SettableFuture (i.e., an asynchronous result of the second signal) class. The general concept of SettableFuture will be briefly introduced below.

[0099] Among them, SettableFuture represents a signal mechanism, which is a derivative of a future object. The future object represents an event that will be completed in the future in asynchronous programming. Assuming that thread A is the completer of the event and thread B is the listener or observer of the event, thread A can check whether the event represented by the future object is completed by calling SettableFuture.isDone; thread A can call SettableFuture.set(Object) to pass in any object, indicating that the event has been completed. Thread A also provides the SettableFuture.get(timeout) interface, and thread B can call the SettableFuture.get(timeout) interface after waiting for timeout (i.e., the consumer driver waits) time to obtain the arbitrary object passed in by thread A, thereby learning that the event has been completed.

[0100] SettableFuture can register a callback function, once the event is completed (i.e. SettableFuture.set is called), the execution body in the callback function is taken out and executed by a certain thread (configurable, generally the thread that calls SettableFuture.set).

[0101] The embodiments of the present disclosure can introduce writable signals and readable signals, and can accurately control the scheduling process of the producer Driver and the consumer Driver according to the memory occupation.

[0102] For the system, the data processing method is provided, which is based on the total byte number of the current data stored in the cache area recorded by the memory counter and the byte number threshold value. When the total byte number of the current data stored in the cache area recorded by the memory counter is greater than the byte number threshold value, the producer Driver is added to the blocking queue to stop writing data into the cache area. When the data in the cache area is read empty or the total byte number of the current data stored in the cache area recorded by the memory counter is 0, the producer Driver is taken out of the blocking queue and added to the ready queue, and the consumer Driver is controlled to be in a waiting state to stop reading data from the cache area. After the consumer Driver waits for a preset time, the time slice is allocated to the producer Driver in the ready queue to make the producer Driver in a running state, so as to wake up the consumer Driver, and then the producer Driver is controlled to write data into the cache area, and the consumer Driver is controlled to read data from the cache area. When the producer Driver is added to the blocking queue, the writable signal can be written into the memory counter, and the callback function registered in the writable signal can be used to take the producer Driver out of the blocking queue and add it to the ready queue when the consumer Driver reads the writable signal. When the data in the cache area is read empty or the total byte number of the current data stored in the cache area recorded by the memory counter is 0, the readable signal can be written into the memory counter, and the callback function registered in the readable signal can be used to wake up the consumer Driver when the producer Driver reads the readable signal, so as to accurately control the scheduling process of the producer Driver and the consumer Driver. In addition, since the data reading speed of the producer Driver is faster than the reading speed of the consumer Driver, when the total byte number of the current data stored in the cache area recorded by the memory counter is greater than the byte number threshold value, the waiting time of the producer Driver is longer. By controlling the producer Driver to be suspended or give up the time slice, the producer Driver can be prevented from waiting in place to occupy system resources and reduce system performance. When the data in the cache area is read empty or the total byte number of the current data stored in the cache area recorded by the memory counter is 0, since the data reading speed of the producer Driver is faster than the reading speed of the consumer Driver, the waiting time of the consumer Driver is shorter. By controlling the consumer Driver to wait in place and giving up the time slice as much as possible, the scheduling overhead can be reduced.

[0103] The data processing method provided by the embodiments of the present disclosure is applied to a system adopting a time slice scheduling strategy and a Pipeline execution framework, and the method is executed by an electronic device. Referring to FIG. 4, the method provided by the embodiments of the present disclosure includes the following steps.

[0104] 401. In any time slice allocated for the consumer driver, when the total number of bytes of data currently stored in the buffer recorded by the memory counter reaches the byte threshold, the producer driver is added to the blocking queue.

[0105] For the system adopting the time slice scheduling strategy and the Pipeline execution framework, the electronic device allocates a time slice for each producer driver (Driver) and each consumer driver (Driver). In the allocated time slice, the producer driver can write data into the buffer, and the consumer driver can read data from the buffer. Since the producer driver gives up the allocated time slice when added to the blocking queue and is allocated a time slice again after being added to the ready queue, that is, the time slice of the producer driver is always dynamically changing, and the consumer driver does not give up the allocated time slice. In order to better illustrate the scheduling process of the producer driver and the consumer driver in a time slice, the embodiments of the present disclosure can take any time slice allocated for the consumer driver as a standard slice for subsequent introduction.

[0106] In any time slice allocated for the consumer driver, whenever the producer driver writes data into the buffer, the first number of bytes of the written data is obtained, and the total number of bytes of data currently stored in the buffer recorded by the memory counter is increased by the first number of bytes. For example, when any producer Driver writes a Chunk on any LocalBuffer, the number of bytes occupied by the Chunk x1 is calculated, and then memory_ledger.addBytes(x1) is executed to increase the total number of bytes recorded by the memory counter by x1.

[0107] In any time slice allocated for the consumer driver, whenever the consumer driver reads data from the buffer, the second number of bytes of the read data is obtained, and the total number of bytes of data currently stored in the buffer recorded by the memory counter is decreased by the second number of bytes. For example, when any producer Driver reads a Chunk from any LocalBuffer, the number of bytes occupied by the Chunk x2 is calculated, and then memory_ledger.subBytes(x2) is executed to decrease the total number of bytes recorded by the memory counter by x2.

[0108] For a system using the Pipeline execution framework, the operation of the producer driver writing data into the buffer area and the operation of the consumer driver reading data from the buffer area can be executed in parallel, so that the total number of bytes recorded by the memory counter is actually the total number of bytes of the actual data stored in the buffer area after the read and write operations at a certain time, for example, at a first time, the total number of bytes recorded by the memory counter is x0, at a second time, the number of bytes of data written into the buffer area by the producer driver is x3, and the number of bytes of data read from the buffer area by the consumer driver is x4, so that the total number of bytes recorded by the memory counter at the second time is x0+x3-x4.

[0109] In the embodiment of the present disclosure, the data writing speed of the producer driver is faster than the data reading speed of the consumer driver, and as the read and write operations proceed, more and more data is stored in the buffer area. To avoid the buffer area storing too much data and causing insufficient system memory, the electronic device will monitor the total number of bytes of data currently stored in the buffer area recorded by the memory counter in the time slice allocated to the consumer driver, and when it is monitored that the total number of bytes of data currently stored in the buffer area recorded by the memory counter reaches the byte threshold value, back pressure will be triggered, the producer driver will give up the allocated time slice, and the producer driver will be added to the blocking queue, so that the producer driver stops writing data. The blocking queue is used to store the producer driver that stops writing data due to giving up the allocated time slice and being in a blocked state, and is also used to store the consumer driver that stops reading data due to giving up the allocated time slice and being in a blocked state. The total number of bytes of data currently stored in the buffer area recorded by the memory counter can reflect the memory occupation, the larger the total number of bytes of data currently stored in the buffer area recorded by the memory counter, the higher the memory occupation rate, and vice versa, the smaller the total number of bytes of data currently stored in the buffer area recorded by the memory counter, the lower the memory occupation rate. In order to be able to visually represent the memory occupation, the total number of bytes of data currently stored in the buffer area recorded by the memory counter can be referred to as the memory water level. Accordingly, the byte threshold value can be referred to as the memory threshold value, which can be defined by the technician, and the embodiment of the present disclosure does not make specific limitations on this. It should be noted that it takes a certain time (usually tens to hundreds of nanoseconds) for data to be read out by the consumer driver and then destroyed and released from the memory, so the total number of bytes of data currently stored in the buffer area recorded in the memory counter, i.e., the memory water level value, cannot reflect the real memory water level at this moment, and the real memory water level is more lagging than the memory water recorded by the memory counter.

[0110] When the producer driver gives up the allocated time slice based on the back pressure mechanism in the time slice corresponding to the consumer driver, the producer driver can be prevented from occupying the time slice, and the system performance can be improved.

[0111] In the embodiments of the present disclosure, when the total number of bytes of data currently stored in the buffer area recorded by the memory counter reaches the byte number threshold, the electronic device further generates a first signal and writes the first signal into the memory counter. The first signal can be a writable signal, and the first signal is used to indicate that a first event is in a waiting state, the first event being an event of writing data into the buffer area by a producer driver. The first signal is registered with a first callback function, and the first callback function is used to put the producer driver into the ready queue from the blocking queue after the consumer driver reads the first signal from the memory counter.

[0112] In the embodiments of the present disclosure, after the producer driver is put into the blocking queue, the producer driver stops writing data into the buffer area, and the operation performed by the consumer driver is not affected. In the time slice allocated for the consumer driver, the electronic device continues to control the consumer driver to read data from the buffer area. Each time the consumer driver reads data from the buffer area, a third number of bytes of the read data is obtained, and then the total number of bytes of data currently stored in the buffer area recorded by the memory counter is reduced by the third number of bytes. After the total number of bytes of data currently stored in the buffer area recorded by the memory counter is reduced by the third number of bytes, and the read data is processed, the data is further cleared from the memory.

[0113] 402、When the total number of bytes of data currently stored in the buffer area recorded by the memory counter is reduced to 0 or the data in the buffer area is read empty, the producer driver is put into the ready queue from the blocking queue, and the consumer driver is controlled to be in a waiting state without giving up the allocated time slice.

[0114] As the consumer driver reads the operation, the data stored in the cache memory becomes less and less, and the total number of bytes of data currently stored in the cache recorded by the memory counter also becomes less and less. When the total number of bytes of data currently stored in the cache recorded by the memory counter decreases to 0 or the data in the cache is read empty, the producer driver needs to continue to write data into the cache to avoid the abnormal data reading of the consumer driver. In order to enable the producer driver to write data into the cache, the electronic device needs to take the producer driver out of the blocking queue and add it to the ready queue. The ready queue is used to store the producer driver or consumer driver in a ready state waiting for allocation of a time slice. Specifically, when the producer driver is taken out of the blocking queue and added to the ready queue, the consumer driver can be controlled to read a first signal from the memory counter. After reading the first signal, a first callback function is called, and the producer driver is taken out of the blocking queue and added to the ready queue. In the embodiment of the present disclosure, the drivers in the ready queue have different priorities, and the drivers in the ready queue are allocated in the order of high to low priority when the time slice is allocated. In order to enable the producer driver taken out of the blocking queue to be preferentially allocated to the time slice to perform the data writing operation, a higher priority can be set for the producer driver. Of course, in order to facilitate the management of producer drivers with different priorities, the ready queue can also be divided into different ready sub-queues, and different ready sub-queues correspond to different priorities and are used to store drivers with different priorities. In order to enable the producer driver taken out of the blocking queue to be preferentially allocated to the time slice, the producer driver taken out of the blocking queue can be added to the ready sub-queue with the highest priority.

[0115] After the producer driver is taken out of the blocking queue and added to the ready queue, the consumer driver is also controlled to be in a waiting state to avoid reading data from the cache. Since the data reading speed of the consumer driver is much lower than the data writing speed of the producer driver, the waiting time of the consumer driver is relatively short, and the scheduling resource overhead of frequent calling of the driver is large. In order to avoid frequent calling of the consumer driver, the embodiment of the present disclosure does not control the consumer driver to give up the time slice when the consumer driver is in the waiting state.

[0116] In this embodiment of the disclosure, when the total number of bytes of data currently stored in the buffer recorded by the memory counter drops to 0 or the data in the buffer is read empty, the electronic device will also generate a second signal and write the second signal into the memory counter. The second signal can be a readable signal, which indicates that a second event is in a waiting state. This second event is the event where the consumer driver reads data from the buffer. The second signal is registered with a second callback function, which is used to wake up the consumer driver after the producer driver reads the second signal.

[0117] 403. After the consumer driver's waiting time reaches the preset time, allocate a time slice to the producer driver in the ready queue so that the producer driver is in the running state, and wake up the consumer driver through the producer driver so that the consumer driver is in the running state.

[0118] The preset duration, denoted as ttimeout, is the maximum duration for which the consumer driver can remain in a waiting state. This preset duration can be set by technical personnel. After the consumer driver's waiting time reaches the preset time, a time slice can be allocated to the producer driver added to the ready queue, allowing the producer driver to enter the running state. While the producer driver is running, it can wake up the consumer driver to enter the running state. Specifically, the producer driver can read a second signal from a memory counter, call a second callback function, and wake up the consumer driver to enter the running state. Considering that the time slice allocated to the consumer driver is finite, and that the consumer driver may remain within the time slice or yield it as time passes, the methods for waking up the consumer driver differ depending on the two scenarios.

[0119] In one possible implementation, if a consumer driver has yielded its time slice and been added to the blocking queue after waiting for a preset time, it is then removed from the blocking queue and added to the ready queue. A new time slice is allocated to the consumer driver added to the ready queue, allowing it to enter the running state. When adding a consumer driver to the ready queue, a higher priority can be set for that consumer driver to ensure it can be allocated a time slice as quickly as possible.

[0120] In another possible implementation, if the consumer driver is within the time slice after a preset time, the consumer driver is directly switched from the waiting state to the running state.

[0121] 404、The producer driver in the running state writes data into the cache area, and the consumer driver in the running state reads data from the cache area.

[0122] When the producer driver is in the running state, the producer driver can be controlled to write data into the cache area until the next back pressure is triggered; when the consumer driver is in the running state, the consumer driver can be controlled to read data from the cache area until the consumer driver gives up the time slice, or the memory counter records that the total number of bytes of data currently stored in the cache area is 0 or the data in the cache area is read empty.

[0123] FIG. 5 shows a flowchart of a data processing method provided by an embodiment of the present disclosure, referring to FIG. 5, the specific process is as follows:

[0124] 1. A memory threshold Smax (i.e. byte threshold) is set in the memory counter (memory_ledger) in advance, and in the time slice allocated for the consumer Driver, the producer Driver writes data Chunk into the cache area, and the consumer Driver reads data Chunk from the cache area. With the read and write operations of the producer Driver and the consumer Driver, when the memory level (i.e. the total number of bytes of data currently stored in the cache area) recorded by the memory counter reaches the memory threshold Smax, the back pressure operation will be triggered, the producer Driver creates a writable signal, and stores the created writable signal in the memory counter.

[0125] 2. The producer Driver gives up the allocated time slice and is suspended in the blocking queue.

[0126] 3. The consumer Driver continues to read data Chunk from the cache area, and at the same time, it also calls memory_ledger.subBytes to reduce the memory level value. When the memory level recorded by the memory counter is reduced to 0 or the cache area queue is empty, the consumer Driver creates a readable signal, and stores the created readable signal in the memory counter.

[0127] 4. The consumer Driver takes out the writable signal from the memory counter, and calls writable.set(Object) (i.e. the first signal (object) set), triggers the callback of the writable signal, to take out the producer Driver from the blocking queue and add it to the highest priority ready queue.

[0128] 5. The producer Driver enters the highest priority ready queue, and waits for the allocated time slice to perform the operation.

[0129] 6. The consumer Driver calls readable.get(timeout) (i.e. get the second signal (timeout)) and the thread starts to block in place, but does not give up the allocated time slice.

[0130] 7. The consumer Driver waits for at most timeout time.

[0131] 8. The producer Driver gets the time slice and performs the operation.

[0132] 9. The producer Driver reads the readable signal from the memory counter and calls readable.set(Object) (i.e. set the second signal (object)) to trigger the callback of the readable signal.

[0133] 10. The producer Driver wakes up the consumer Driver to continue the read operation. If the timeout time is exceeded, the consumer Driver gives up the time slice and enters the blocking queue, then the consumer Driver is taken out of the blocking queue and added to the highest priority ready queue, and then the time slice is allocated to the consumer Driver; if the timeout time is not exceeded, the consumer Driver is directly switched from the waiting state to the running state. Then, the producer Driver starts to produce the Chunk and write it into the cache area LocalBuffer, the consumer Driver continues to read the Chunk from the cache area, processes the read Chunk, and then destroys the memory occupied by the Chunk.

[0134] Considering that the scheduling of the Driver itself has overhead in the time slice scheduling mechanism, and the scheduling of the involved producer Driver is needed every time back pressure occurs. The overhead includes the following parts: when the producer Driver gives up the time slice to enter the blocking or ready queue, the partial intermediate state of the producer Driver (such as context information) needs to be saved, and the current instruction and data need to be cached to avoid cache misses; when the producer Driver enters the running state from the ready queue, the context information of the producer Driver needs to be reloaded, and cache misses also occur during running. The above process is similar to the context switch process in the operating system. In the actual measurement of a database product instance, it is found that the scheduling overhead takes about several hundred microseconds; and if the scheduling frequency of a producer Driver exceeds 1000 times / s, or the total scheduling time exceeds 100 ms within 1 s, a significant performance regression will occur. Therefore, to improve system performance, the total back pressure frequency within a time slice needs to be strictly controlled to ensure that the total back pressure frequency is within a reasonable threshold. In order to strictly control the total back pressure frequency within a time slice, the following first analyzes the influencing factors of the total back pressure frequency within a time slice in combination with the read-write operation process of the producer Driver and the consumer Driver within a standard time slice.

[0135] In the back pressure execution flow, the memory counter memory_ledger records the number of bytes written by the producer Driver and the number of bytes read by the consumer Driver, and records the time length of each operation of the producer Driver and the consumer Driver. Through calculation, it can be obtained that the speed of the producer Driver writing data into the cache area is V_1, and the speed of the consumer Driver reading data from the cache area is V_2. Assuming that T is a standard time slice length, T1 is the time from the start of the producer Driver writing data into the cache area to triggering back pressure, R is the total number of times of scheduling of the producer Driver or the total number of times of back pressure within a standard time slice, let t_timeout be the time for the consumer Driver to wait for the readable signal to trigger the callback, S be the memory level of the cache area recorded by the memory counter memory_ledger at a certain time, and S_max be the memory threshold. In a standard time slice, assuming that the consumer Driver does not give up the time slice at all, the change rule of the memory level S of the cache area with time t is as shown in FIG. 6.

[0136] Referring to FIG. 6, initially, the producer Driver and the consumer Driver run simultaneously, the producer Driver writes data into the cache area at a speed of V_1, and the consumer Driver reads data from the cache area at a speed of V_2. At T1, the memory level recorded by the memory counter reaches the set memory threshold S_max, and thus T_1=S_max / (V_1-V_2) can be obtained.

[0137] At T1, the total number of bytes written by the producer Driver is T_1V_1.

[0138] After T1, the producer Driver enters the blocking queue due to the back pressure mechanism and waits, while the consumer Driver continues to read data from the cache area until the memory level recorded by the memory counter is 0 or the cache area queue is empty. At this time, the producer Driver is woken up and waits for at most t_timeout. The producer Driver is woken up and starts to write data into the cache area, at this time, the consumer Driver is also woken up and starts to read data from the cache area, and the above steps are repeatedly executed until a standard time slice ends. Since the consumer Driver is in a waiting state for a part of the time, and the waiting time is tens of nanoseconds each time, therefore, the upper limit of the data read by the consumer Driver from the cache area in a standard time slice T is ≤V_2T.

[0139] Therefore, the total back pressure times in a standard time slice T is R≤(V_2T) / (T_1V_1)=(V_2(V_1-V_2)) / V_1 T / S_max.

[0140] It can be analyzed that, in a given standard time slice length T, the value of T1 is unstable, but the above formula avoids the influence caused by the unstable value of T1. The total back pressure times R is related to the speed V_1 at which the producer Driver writes data into the cache area, the speed V_2 at which the consumer Driver reads data from the cache area, and the set memory threshold S_max. However, in a standard time slice length, the speed V_1 at which the producer Driver writes data into the cache area and the speed V_2 at which the consumer Driver reads data from the cache area are usually fixed, and thus, if it is desired to ensure that the total back pressure times does not exceed the back pressure times threshold, it is necessary to adjust the memory threshold S_max. The back pressure times threshold can be the maximum total back pressure times that can be tolerated in a time slice, and the back pressure times threshold can be 800, 1000, etc. The default back pressure times threshold of the system is usually 1000. Considering that different users have different requirements for system performance, a database parameter interface can also be reserved for users to adjust.

[0141] Based on the above principle, before performing the data read-write operation within the time slice allocated for the consumer driver program, it can be judged based on the first speed of the producer driver program writing data, the second speed of the consumer driver program reading data and the initial byte number threshold value whether the set byte number threshold value can guarantee that the total back pressure number does not exceed the back pressure number threshold value. If it cannot be guaranteed, the set initial byte number threshold value needs to be adjusted until the set byte number threshold value can guarantee that the total back pressure number does not exceed the back pressure number threshold value. The specific judgment process includes:

[0142] First, based on the first speed of the producer driver program writing data, the second speed of the consumer driver program reading data and the initial byte number threshold value, the total back pressure number of the producer driver program within the time slice is calculated.

[0143] Wherein, the initial byte number threshold value can be 4MB, 8MB, etc., and the system is usually defaulted to 8MB. Specifically, based on the first speed of the producer driver program writing data, the second speed of the consumer driver program reading data and the initial byte number threshold value, the total back pressure number of the producer driver program within the time slice is calculated, including:

[0144] 1) Based on the first speed, the second speed and the initial byte number threshold value, the first back pressure time of the producer driver program is calculated.

[0145] Assuming that the first speed is V_1, the second speed is V_2, and the initial byte number threshold value is S_max, then the first back pressure time T1 is: T_1=S_max / (V_1-V_2)

[0146] 2) Based on the first speed and the first back pressure time, the first total byte number of the producer driver program writing data when reaching the first back pressure time is calculated.

[0147] By multiplying the first speed and the first back pressure time, the first total byte number of the producer driver program writing data when reaching the first back pressure time can be obtained as T_1V_1.

[0148] 3) Based on the length of the time slice and the second speed, the second total byte number of the consumer driver program reading data within the time slice is calculated.

[0149] Considering that there is a part of time in a time slice length that the consumer Driver is in a waiting state, then the second total byte number of the consumer driver program reading data within the time slice is ≤V_2T.

[0150] 4) The ratio of the first total byte number and the second total byte number is calculated to obtain the total back pressure number.

[0151] The total anti-pressure times R can be obtained by calculating the ratio of the first total byte number and the second total byte number: R≤(V_2T) / (T_1V_1)=(V_2(V_1-V_2)) / V_1 T / S_max

[0152] In the second step, when the total anti-pressure times exceeds the anti-pressure times threshold, the initial byte number threshold is adjusted to obtain a byte number threshold, which can make the calculated total anti-pressure times not exceed the anti-pressure times threshold.

[0153] When the total anti-pressure times exceeds the anti-pressure times threshold, the initial byte number threshold is continuously adjusted, and the total anti-pressure times is calculated based on the adjusted byte number threshold by using the method in the first step. If the calculated total anti-pressure times does not exceed the anti-pressure times threshold, the adjusted byte number threshold corresponding to the total anti-pressure times is taken as the byte number threshold corresponding to the time slice.

[0154] In the length of the time slice, the determined byte number threshold is unchanged, and in the length of the next time slice, the byte number threshold can be adjusted. In the actual application of the database instance, it is found that V_1 and V_2 are relatively stable in a standard time slice, and after the value of S_max is adjusted for the first time, the number of times of triggering the dynamic adjustment of S_max is not large.

[0155] In addition, if the set anti-pressure times threshold is too small, S_max may take a value that is too large, which will also cause the problem of out of memory, and make the anti-pressure invalid. Therefore, in the actual application, an upper limit value of S_max itself can also be set according to the size of the memory of the database instance when running.

[0156] Referring to FIG. 7, after S_max is set and V_1 and V_2 are calculated, in any time slice set for the consumer Driver, the producer Driver writes data into the cache area at a speed of V_1, and the consumer reads data from the cache area at a speed of V_2. In the process of reading and writing speeds of the producer Driver and the consumer Driver, the memory counter records the total byte number of the data currently stored in the cache area, to control the scheduling of the producer Driver and the consumer Driver.

[0157] The embodiment of the present disclosure is directed to a system adopting a time slice scheduling strategy and a Pipeline execution framework, and can perform data processing based on a back pressure mechanism in the case that the read-write speed of a producer Driver and a consumer Driver is greatly different and the buffer memory is limited. In addition, a logical buffer area is constructed, and a lighter memory counting method is used for counting, so as to avoid the overhead of serialization and deserialization and memory copying. In addition, based on the time slice scheduling mechanism, the read-write speed of the producer Driver and the consumer Driver is measured, and the memory threshold is adjusted to control the back pressure frequency, so as to avoid frequent context switching and improve the overall performance.

[0158] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0159] Please refer to FIG. 8, which shows a structural schematic diagram of a data processing apparatus provided by the embodiment of the present disclosure. The apparatus is applied to a system adopting a time slice scheduling strategy and a Pipeline execution framework. The apparatus can be realized by software, hardware or a combination of both, and become all or part of an electronic device. The apparatus includes:

[0160] The first joining module 801 is configured to, when the total number of bytes of data currently stored in the buffer area recorded by the memory counter decreases to 0 or the data in the buffer area is read empty in any time slice allocated for the consumer Driver, take the producer Driver out of the blocked queue and join the ready queue, the blocked queue being used to store the producer Driver stopped from writing data due to giving up the allocated time slice in a blocked state, and the ready queue being used to store the producer Driver or consumer Driver in a ready state waiting for allocation of a time slice.

[0161] The first control module 802 is configured to control the consumer Driver to be in a waiting state without giving up the allocated time slice.

[0162] The allocation module 803 is configured to allocate a time slice to the producer Driver in the ready queue after the waiting duration of the consumer Driver reaches a preset duration, so that the producer Driver is in a running state.

[0163] The wake-up module 804 is configured to wake up the consumer Driver by the producer Driver, so that the consumer Driver is in a running state.

[0164] The second control module 805 is configured to control the producer Driver in the running state to write data into the buffer area.

[0165] The third control module 806 is configured to control the consumer Driver in the running state to read data from the buffer area.

[0166] In another embodiment of the present disclosure, the apparatus further comprises:

[0167] The first obtaining module is configured to, in the time slice allocated for the consumer driver, obtain a first number of bytes of data written by the producer driver into the cache area each time the producer driver writes data into the cache area;

[0168] The first recording module is configured to increase, by the first number of bytes, the total number of bytes of data currently stored in the cache area recorded by the memory counter;

[0169] The second obtaining module is configured to obtain a second number of bytes of data read by the consumer driver from the cache area each time the consumer driver reads data from the cache area;

[0170] The second recording module is configured to decrease, by the second number of bytes, the total number of bytes of data currently stored in the cache area recorded by the memory counter;

[0171] The second adding module is configured to perform the operation of adding the producer driver into the blocking queue when the total number of bytes of data currently stored in the cache area recorded by the memory counter reaches the number-of-bytes threshold.

[0172] In another embodiment of the present disclosure, the apparatus further comprises:

[0173] The first writing module is configured to write a first signal into the memory counter when the total number of bytes of data currently stored in the cache area recorded by the memory counter reaches the number-of-bytes threshold, the first signal being used to indicate that a first event is in a waiting state, the first event being an event of the producer driver writing data into the cache area, the first signal being registered with a first callback function, the first callback function being used to add the producer driver into the ready queue from the blocking queue after the consumer driver reads the first signal from the memory counter.

[0174] In another embodiment of the present disclosure, the second adding module is configured to control the consumer driver to read the first signal from the memory counter, and call the first callback function to add the producer driver into the ready queue from the blocking queue after reading the first signal.

[0175] In another embodiment of the present disclosure, the apparatus further comprises:

[0176] The fourth control module is configured to control the consumer driver to read data from the cache area in the time slice allocated for the consumer driver after the producer driver is added into the blocking queue.

[0177] The third obtaining module is configured to obtain a third number of bytes of data read by the consumer driver from the cache area each time the consumer driver reads data from the cache area;

[0178] The third recording module is configured to reduce the total number of bytes of the current stored data of the buffer recorded by the memory counter by a third number of bytes until the total number of bytes of the current stored data of the buffer recorded by the memory counter is reduced to 0 or the data of the buffer is read empty.

[0179] In another embodiment of the present disclosure, the apparatus further comprises:

[0180] The second writing module is configured to write a second signal into the memory counter when the total number of bytes of the current stored data of the buffer recorded by the memory counter is reduced to 0 or the data of the buffer is read empty, the second signal being used to indicate that a second event is in a waiting state, the second event being an event of reading data from the buffer by a consumer driver, the second signal being registered with a second callback function, the second callback function being used to wake up the consumer driver after the producer driver reads the second signal.

[0181] In another embodiment of the present disclosure, the waking-up module is configured to control the producer driver to read the second signal from the memory counter, to call the second callback function, and to wake up the consumer driver so as to make the consumer driver in a running state.

[0182] In another embodiment of the present disclosure, the waking-up module is configured to, if the consumer driver has given up a time slice and is added to a blocking queue after waiting for a preset time, take the consumer driver out of the blocking queue and add the consumer driver to a ready queue, to allocate a new time slice to the consumer driver in the ready queue, and to make the consumer driver in a running state; and if the consumer driver is in the time slice after the preset time, switch the consumer driver from the waiting state to the running state.

[0183] In another embodiment of the present disclosure, the apparatus further comprises:

[0184] The calculating module is configured to calculate a total number of times of back pressure of the producer driver in the time slice based on the first speed of writing data by the producer driver, the second speed of reading data by the consumer driver, and the initial number of bytes threshold.

[0185] The adjusting module is configured to, when the total number of times of back pressure exceeds the number of times of back pressure threshold, adjust the initial number of bytes threshold to obtain a number of bytes threshold, the number of bytes threshold being able to make the calculated total number of times of back pressure not exceed the number of times of back pressure threshold.

[0186] In another embodiment of the present disclosure, the computing module is configured to calculate a first back pressure time of the producer driver based on the first speed, the second speed and the initial byte number threshold; calculate a first total byte number of data written by the producer driver when the first back pressure time is reached based on the first speed and the first back pressure time; calculate a second total byte number of data read by the consumer driver within the time slice based on the length of the time slice and the second speed; and calculate a ratio of the first total byte number and the second total byte number to obtain the total back pressure times.

[0187] FIG. 9 shows a structural block diagram of an electronic device 900 according to an example embodiment of the present disclosure. Generally, the electronic device 900 includes a processor 901 and a memory 902.

[0188] The processor 901 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 901 can also include a main processor and a co-processor, the main processor being a processor for processing data in an awake state, and the co-processor being a low-power processor for processing data in a standby state. In some embodiments, the processor 901 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content to be displayed on a display screen. In some embodiments, the processor 901 can further include an artificial intelligence processor for processing machine learning-related computing operations.

[0189] The memory 902 can include one or more computer readable storage media that can be non-transitory computer readable storage media, such as CD-ROM (Compact Disc Read-Only Memory), ROM (Read-Only Memory), RAM (Random Access Memory), magnetic tape, floppy disk, and optical data storage devices, etc. The computer readable storage media stores at least one computer program that is executable when executed to implement the data processing method.

[0190] Of course, the electronic device described above can also include other components, such as an input / output interface, a communication component, and the like. The input / output interface provides an interface between the processor and a peripheral interface module, which can be an output device, an input device, and the like. The communication component is configured to facilitate wired or wireless communication between the electronic device and other devices.

[0191] Those skilled in the art can understand that the structure shown in FIG. 9 does not constitute a limitation on the electronic device 900, and can include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0192] The embodiments of the present disclosure provide a computer readable storage medium, the computer readable storage medium stores at least one computer program, and the at least one computer program can implement the data processing method when executed by a processor.

[0193] The embodiments of the present disclosure provide a computer program product, the computer program product includes a computer program, and the computer program can implement the data processing method when executed by a processor.

[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0195] The above embodiments are only used to illustrate the technical solutions of the present disclosure, but not limit them; although the foregoing embodiments of the present disclosure are described in detail, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A data processing method, characterized by, The method is applied to a system adopting a time slice scheduling strategy and a Pipeline execution framework, and the method comprises the following steps: In any time slice allocated for a consumer driver, when the total number of bytes of data currently stored in a cache area recorded by a memory counter decreases to 0 or the data of the cache area is read empty, a producer driver is taken out of a blocking queue and added to a ready queue, the blocking queue is used to store the producer driver which stops writing data in a blocked state due to giving up the allocated time slice, and the ready queue is used to store the producer driver in a ready state waiting for the allocation of a time slice, and the consumer driver is controlled to be in a waiting state without giving up the allocated time slice. After the waiting duration of the consumer driver reaches a preset duration, a time slice is allocated to the producer driver in the ready queue, so that the producer driver is in a running state, and the consumer driver is woken up by the producer driver, so that the consumer driver is in a running state. The producer driver in the running state is controlled to write data into the cache area, and the consumer driver in the running state is controlled to read data from the cache area.

2. The method of claim 1, wherein, The method further comprises the following steps: In the time slice allocated for the consumer driver, whenever the producer driver writes data into the cache area, a first number of bytes of the written data is obtained, and the total number of bytes of data currently stored in the cache area recorded by the memory counter is increased by the first number of bytes. Whenever the consumer driver reads data from the cache area, a second number of bytes of the read data is obtained, and the total number of bytes of data currently stored in the cache area recorded by the memory counter is decreased by the second number of bytes. When the total number of bytes of data currently stored in the cache area recorded by the memory counter reaches a byte number threshold, the producer driver is added to the blocking queue.

3. The method according to claim 1 or 2, characterized in that, The method further comprises the following steps: When the total number of bytes of data currently stored in the cache area recorded by the memory counter reaches the byte number threshold, a first signal is written into the memory counter, the first signal is used to indicate that a first event is in a waiting state, the first event is the event that the producer driver writes data into the cache area, the first signal is registered with a first callback function, and the first callback function is used to take the producer driver out of the blocking queue and add it to the ready queue after the consumer driver reads the first signal from the memory counter.

4. The method of claim 3, wherein, The taking of the producer driver out of the blocking queue and adding it to the ready queue comprises the following steps: The consumer driver is controlled to read the first signal from the memory counter, and after reading the first signal, the first callback function is called to take the producer driver out of the blocking queue and add it to the ready queue.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises the following steps: When the producer driver is added to the blocking queue, the consumer driver is controlled to read data from the cache area within the time slice allocated to the consumer driver, and each time the consumer driver reads data from the cache area, the third number of bytes of the read data is obtained, the total number of bytes of data currently stored in the cache area recorded by the memory counter is reduced by the third number of bytes, until the total number of bytes of data currently stored in the cache area recorded by the memory counter is reduced to 0 or the data in the cache area is read empty.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: When the total number of bytes of data currently stored in the cache area recorded by the memory counter is reduced to 0 or the data in the cache area is read empty, a second signal is written into the memory counter, the second signal is used to indicate that a second event is in a waiting state, the second event is an event of reading data from the cache area by the consumer driver, and the second signal is registered with a second callback function, the second callback function is used to wake up the consumer driver after the producer driver reads the second signal.

7. The method of claim 6, wherein, The method further comprises: The producer driver is controlled to read the second signal from the memory counter, the second callback function is called, and the consumer driver is woken up to make the consumer driver in a running state.

8. The method of claim 7, wherein, The method further comprises: If the consumer driver has given up the time slice after waiting for the preset time and the consumer driver is added to the blocking queue, the consumer driver is taken out of the blocking queue and added to the ready queue, a new time slice is allocated to the consumer driver in the ready queue, and the consumer driver is in a running state, the blocking queue is also used to store the consumer driver that stops reading data in a blocked state due to giving up the allocated time slice, and the ready queue is also used to store the consumer driver in a ready state waiting for allocation of a time slice; If the consumer driver is in the time slice after the preset time, the consumer driver is switched from a waiting state to a running state.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: Based on the first speed of the producer driver writing data, the second speed of the consumer driver reading data, and an initial byte number threshold, a total number of times of back pressure of the producer driver within the time slice is calculated; When the total number of times of back pressure exceeds a back pressure number threshold, the initial byte number threshold is adjusted to obtain the byte number threshold, and the byte number threshold can make the calculated total number of times of back pressure not exceed the back pressure number threshold.

10. The method of claim 9, wherein, The method further comprises: The total number of times of back pressure of the producer driver within the time slice is calculated based on the first speed of the producer driver writing data, the second speed of the consumer driver reading data, and an initial byte number threshold. calculating a first back pressure time of the producer driver based on the first speed, the second speed and the initial byte threshold; calculating a first total byte number of data written by the producer driver when the first back pressure time is reached based on the first speed and the first back pressure time; calculating a second total byte number of data read by the consumer driver within the time slice based on the length of the time slice and the second speed; calculating a total back pressure number by dividing the first total byte number by the second total byte number.

11. A data processing apparatus, characterized by The device is applied to a system adopting a time slice scheduling strategy and a Pipeline execution framework, and the device comprises: a first joining module, configured to, when a total byte number of data currently stored in a buffer recorded by a memory counter falls to 0 or data in the buffer is read empty within any time slice allocated to a consumer driver, take out a producer driver from a blocked queue of the producer driver and add the producer driver to a ready queue, the blocked queue being used to store the producer driver which stops writing data due to being blocked for giving up the allocated time slice, and the ready queue being used to store the producer driver which is in a ready state and waits for allocation of a time slice; a first control module, configured to control the consumer driver to be in a waiting state and not to give up the allocated time slice, an allocation module, configured to allocate a time slice to the producer driver in the ready queue after a waiting duration of the consumer driver reaches a preset duration, so that the producer driver is in a running state; a wake-up module, configured to wake up the consumer driver by the producer driver, so that the consumer driver is in the running state; a second control module, configured to control the producer driver in the running state to write data into the buffer; a third control module, configured to control the consumer driver in the running state to read data from the buffer.

12. An electronic device, comprising: The device comprises a processor and a memory, the memory stores at least one program code, and the at least one program code is used to be called and executed by the processor to implement the data processing method in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, and the at least one computer program is executed by the processor to implement the data processing method in any one of claims 1 to 10.

14. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the data processing method in any one of claims 1 to 10.

Images