Streaming task operation method and device based on distributed batch scene

Abstract

The invention discloses a streaming task operation method and device based on a distributed batch scene, and relates to the technical field of distribution.The method comprises the steps that after a task scheduling instruction distributed by a scheduler is received, the global streaming read thread count in a distributed cache is inquired, and when the task scheduling instruction does not exceed a preset thread count threshold value, the task scheduling instruction is executed; adding the job task into a streaming read task execution queue, scheduling a to-be-executed task from the streaming read task execution queue under the condition of monitoring that the target cache triggers a resource release event, confirming a target thread corresponding to the to-be-executed task, and reading task data from a database through a streaming read mode according to the to-be-executed task; after task processing is completed, thread resources corresponding to the target thread are released, and the thread occupancy state in the target cache is updated. According to the method and the device, the technical problems of long task processing time, excessive consumption of thread resources and influence on database use due to adoption of a streaming reading strategy in a distributed batch scene in related technologies are solved.

Description

Technical Field

[0001] The present invention relates to the field of distributed technology, big data technology or other related fields, and in particular to a streaming task operation method and device based on a distributed batch scenario. Background Art

[0002] In large-scale data processing and distributed computing environments, batch data reading operations pose a severe challenge to database systems. This is especially true when dealing with large datasets. Traditional data retrieval methods can severely degrade database or application performance, even leading to "Out of Memory" (OOM) errors. This can cause instability in production environments and impact the normal operation of online transaction services. Currently, the industry generally adopts streaming read strategies as an effective solution for large-volume data reading. These strategies allow databases to incrementally send data to clients rather than loading all data at once, effectively reducing memory pressure on both the client and server sides.

[0003] However, the streaming read mode has some limitations in practical applications. For example, streaming read requests require constant communication with the client during data processing. Based on the consumption of the client's socket cache, new data is sent to the client at irregular intervals. This requires the thread usage rights to be continuously occupied and will not be released until the streaming read is completed. When the data volume is large enough and the number of shards exceeds the size of the database's internal thread pool, the continuous streaming read operation will occupy thread resources for a long time, and the thread usage rights will not be released until the streaming read is completed. This means that if the streaming read task processing time is too long, it may exhaust all available threads in the thread pool, making the database service unable to respond to other requests, seriously affecting its availability and responsiveness, especially when the database needs to support a large number of concurrent online transactions.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present invention provide a streaming task operation method and device based on a distributed batch scenario, so as to at least solve the technical problems in the related art that the use of a streaming reading strategy in a distributed batch scenario is prone to long task processing time, excessive consumption of thread resources, and impact on database usage.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a streaming task operation method based on a distributed batch scenario is provided, which is applied to executors distributedly deployed in multiple campus centers, and all of the executors share the global streaming read thread number through the target cache, including: after receiving the task scheduling instruction distributed by the scheduler, querying the global streaming read thread number in the distributed cache; when the global streaming read thread number does not exceed the preset thread number threshold, adding the job task in the task scheduling instruction to the streaming read task execution queue; when monitoring the target cache triggering resource release event, scheduling the task to be executed from the streaming read task execution queue, and confirming the target thread corresponding to the task to be executed; according to the task to be executed, reading the task data from the database through the streaming read mode; after completing the task processing, releasing the thread resources corresponding to the target thread, and updating the thread occupancy status in the target cache.

[0007] Optionally, before receiving the task scheduling instruction distributed by the scheduler, it includes: using a global counter to count the number of threads currently executing the task to obtain the global number of streaming reading threads, wherein, during the counting process, a preset distributed atomic operation command is used to perform statistical conflict avoidance operations on the modification of the same data in the database, and the conflict avoidance operation means that there is only one modification operation for the modification of the same data at the same time point; based on the statistically obtained number of global streaming reading threads, the thread occupancy status is updated.

[0008] Optionally, before receiving the task scheduling instruction distributed by the scheduler, it also includes: the scheduler obtains the executor node status through the distributed application coordination service center; the scheduler divides the task into shards based on the job definition and task metadata in the database, and generates the task scheduling instruction, wherein the task metadata includes at least one of the following: the shard to which the task belongs, the task priority, and the task timeout duration; the scheduler distributes the task scheduling instruction to the executors deployed in the multi-campus center through the message queue.

[0009] Optionally, the step of the scheduler distributing the task scheduling instructions to the executors deployed in the multi-campus centers through the message queue includes: the scheduler calculates the shard hash value based on the shard to which the task belongs; the scheduler distributes the task scheduling instructions to the executors deployed in the multi-campus centers through the message queue based on the shard hash value and the task priority.

[0010] Optionally, according to the task to be executed, the step of reading task data from the database through the streaming reading mode includes: counting the execution time of the task to be executed; when the execution time exceeds the preset execution time threshold, confirming that the task execution is abnormal, and forcibly releasing the target thread through a scheduled job to prevent deadlock.

[0011] Optionally, after completing task processing, the thread resources corresponding to the target thread are released, and the thread occupancy status in the target cache is updated, including: when releasing the thread resources corresponding to the target thread, obtaining other tasks belonging to the same slice as the task to be executed; batch pulling the thread resources released by the task to be executed and the other tasks belonging to the same slice, and updating the thread occupancy status in the target cache.

[0012] Optionally, after querying the global number of streaming read threads in the distributed cache, it also includes: when the global number of streaming read threads exceeds a preset thread number threshold, adding the task scheduling instruction to the streaming read task waiting queue, wherein the queue length of the streaming read task waiting queue is dynamically adjusted according to the remaining thread resources of the database.

[0013] According to another aspect of an embodiment of the present invention, a streaming task operation device based on a distributed batch scenario is also provided, which is applied to executors distributedly deployed in multiple campus centers, and all of the executors share a global streaming read thread number through a target cache, including: a thread number query unit, which is used to query the global streaming read thread number in the distributed cache after receiving a task scheduling instruction distributed by a scheduler; a task queue joining unit, which is used to add the job task in the task scheduling instruction to the streaming read task execution queue when the global streaming read thread number does not exceed a preset thread number threshold; a task scheduling unit, which is used to schedule tasks to be executed from the streaming read task execution queue and confirm the target thread corresponding to the task to be executed when a target cache triggers a resource release event; a task execution unit, which is used to read task data from a database through a streaming read mode according to the task to be executed; a thread status update unit, which is used to release the thread resources corresponding to the target thread after completing task processing, and update the thread occupancy status in the target cache.

[0014] Optionally, the streaming task operation device based on a distributed batch scenario also includes: a counting unit, which is used to use a global counter to count the number of threads currently executing the task before receiving the task scheduling instruction distributed by the scheduler to obtain the global streaming reading thread number, wherein, during the counting process, a preset distributed atomic operation command is used to perform statistical conflict avoidance operations on the modification of the same data in the database, and the conflict avoidance operation means that there is only one modification operation for the modification of the same data at the same time point; a thread occupancy status update unit, which is used to update the thread occupancy status based on the global streaming reading thread number obtained by statistics.

[0015] Optionally, the streaming task operation device based on a distributed batch scenario also includes: a node status acquisition unit, which is used for the scheduler to obtain the executor node status through the distributed application coordination service center before receiving the task scheduling instruction distributed by the scheduler; a slicing unit, which is used for the scheduler to divide the task into slicings based on the job definition and task metadata in the database, and generate the task scheduling instruction, wherein the task metadata includes at least one of the following: the slicing to which the task belongs, the task priority, and the task timeout duration; an instruction distribution unit, which is used for the scheduler to distribute the task scheduling instruction to the executors deployed in the multi-campus center through a message queue.

[0016] Optionally, the instruction distribution unit includes: a shard hash value calculation module, which is used by the scheduler to calculate the shard hash value based on the shard to which the task belongs; an instruction distribution module, which is used by the scheduler to distribute the task scheduling instruction to the executors deployed in the multi-park center through a message queue based on the shard hash value and the task priority.

[0017] Optionally, the task execution unit includes: a task execution time statistics module, used to count the execution time of the task to be executed; a task exception handling module, used to confirm the task execution exception when the execution time exceeds the preset execution time threshold, and forcibly release the target thread through a timed job to prevent deadlock.

[0018] Optionally, the thread status update unit includes: other same-slice task acquisition module, used to acquire other tasks belonging to the same slice as the task to be executed when releasing the thread resources corresponding to the target thread; a thread resource batch pulling module, used to batch pull the thread resources released by the task to be executed and other tasks belonging to the same slice, and update the thread occupancy status in the target cache.

[0019] Optionally, the streaming task operation device based on the distributed batch scenario also includes: a waiting queue joining unit, which is used to add the task scheduling instruction to the streaming reading task waiting queue after querying the global streaming reading thread number in the distributed cache, when the global streaming reading thread number exceeds the preset thread number threshold, wherein the queue length of the streaming reading task waiting queue is dynamically adjusted according to the remaining thread resources of the database.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned streaming task operation methods based on distributed batch scenarios.

[0021] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned streaming task operation methods based on distributed batch scenarios.

[0022] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements the steps of any one of the above-mentioned methods for stream task operation based on a distributed batch scenario.

[0023] In the present disclosure, after receiving the task scheduling instruction distributed by the scheduler, the global number of streaming read threads in the distributed cache is queried; when the global number of streaming read threads does not exceed the preset thread number threshold, the job task in the task scheduling instruction is added to the streaming read task execution queue; when the target cache triggers a resource release event, the task to be executed is scheduled from the streaming read task execution queue, and the target thread corresponding to the task to be executed is confirmed; according to the task to be executed, the task data is read from the database through the streaming read mode; after completing the task processing, the thread resources corresponding to the target thread are released, and the thread occupancy status in the target cache is updated.

[0024] Based on the above-mentioned public content, the global number of streaming reading threads can be compared with the preset thread number threshold, and the comparison result can be used to confirm whether the job task is added to the streaming reading task execution queue to avoid excessive consumption of thread resources and affect the execution of other important tasks. At the same time, after completing the task processing, the thread resources corresponding to the target thread are released in time, and the thread occupancy status in the target cache is updated, so that the thread resources can be maintained in a reasonable state, thereby solving the technical problems in related technologies that the use of streaming reading strategies in distributed batch scenarios is prone to long task processing time, excessive consumption of thread resources, and impact on database usage.

[0025] Furthermore, by dynamically adjusting the capacity and priority scheduling of the streaming read task waiting queue, the present invention avoids excessive competition for database thread pool resources, ensuring a balance between batch processing and online transactions, and reducing the risk of database services becoming unavailable due to resource exhaustion. Furthermore, the present invention utilizes a multi-shard concurrent streaming read mechanism to rationally distribute database read tasks, reducing the continuous resource occupation of individual tasks and improving concurrent processing capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a streaming task operation method based on a distributed batch scenario is shown;

[0028] Figure 2 This is a flowchart of an optional streaming task operation method based on a distributed batch scenario according to an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of thread scheduling during the interaction between each application container and the database in an optional distributed batch scenario according to an embodiment of the present invention;

[0030] Figure 4 This is a system architecture diagram for implementing multi-shard job tasks in an optional distributed batch scenario according to an embodiment of the present invention;

[0031] Figure 5 is a schematic diagram of an optional streaming task operation device based on a distributed batch scenario according to an embodiment of the present invention;

[0032] Figure 6 This is a structural block diagram of an electronic device for executing a streaming task operation method based on a distributed batch scenario according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] To facilitate those skilled in the art to understand the present invention, some of the terms or nouns involved in the embodiments of the present invention are explained below:

[0036] Distributed Cache is a storage system that shares data across multiple server nodes, enabling fast data access, reducing direct database requests, and improving system responsiveness and overall performance. In this paper, the distributed cache is used to monitor and manage database thread pool usage in real time, ensuring a controllable number of concurrent streaming read operations and avoiding resource conflicts.

[0037] Streaming Read is a data reading mode that allows the database to send data in batches instead of loading all data into the client memory at once.

[0038] Redis, or Remote Dictionary Server, is an open-source, high-performance key-value store that supports a variety of data structures, such as strings, hashes, lists, and sets. In this paper, Redis is used as a distributed cache. Through its atomic operations and message subscription capabilities, it enables real-time monitoring of database resource usage and efficient scheduling of streaming read tasks.

[0039] PUB / SUB (Publish / Subscribe) is a messaging model that allows publishers to publish messages to specific topics, while subscribers subscribe to these topics and receive and consume messages sent by publishers. In this paper, the PUB / SUB mechanism is used to detect thread resource changes in real time in Redis, triggering task wakeup and scheduling.

[0040] The queue executor is a component in a distributed system that is responsible for receiving, queuing, and executing tasks. In the present invention, the queue executor places streaming read tasks into a waiting queue according to the instructions of the scheduler and schedules them for execution according to priority when resources are available.

[0041] Controller, plays a coordination and management role in distributed systems. It is responsible for receiving requests, generating scheduling instructions and managing the activities of executor nodes.

[0042] It should be noted that the streaming task operation method and device based on distributed batch scenarios in the present disclosure can be used in the field of distributed technology to implement streaming reading of multi-shard jobs based on distributed batch scenarios, and can also be used in any field other than the field of distributed technology to implement streaming reading of multi-shard jobs based on distributed batch scenarios. The present disclosure does not limit the application field of the streaming task operation method and device based on distributed batch scenarios.

[0043] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0044] It should be noted that in this disclosure, when collecting and analyzing customer information, the corresponding operation entrance is provided for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0045] The following embodiments of the present invention can be applied to various systems, applications, and devices for streaming tasks in distributed batch scenarios. The present invention can be applied to distributed computing scenarios and large-scale batch processing scenarios, such as large-scale data processing in the financial industry. These scenarios include batch data migration, historical account audits, and customer data analysis within financial institutions. These scenarios process enormous amounts of data, placing extremely high demands on database read performance, which can easily lead to resource contention and unstable database services.

[0046] By dynamically adjusting the capacity and priority scheduling of the streaming read task waiting queue, this invention avoids excessive competition for database thread pool resources, ensuring a balance between batch processing and online transactions, and reducing the risk of database services becoming unavailable due to resource exhaustion. Furthermore, this invention utilizes a multi-shard concurrent streaming read mechanism to rationally distribute database read tasks, reducing the continuous resource occupation of individual tasks and improving concurrent processing capabilities.

[0047] Furthermore, the present invention introduces a distributed cache to monitor the usage of the database thread pool. Combined with dynamic threshold control and resource release event monitoring, this allows for refined management of streaming read thread resources. This allows for automatic adjustment of job scheduling strategies based on real-time system resource status, improving the overall system's response speed and throughput. Furthermore, the present invention provides a timed job and resource release trigger mechanism that ensures streaming read tasks are quickly dispatched when resources become available, reducing task wait times. Furthermore, by optimizing the concurrent performance of streaming reads, data processing delays are reduced, improving the overall efficiency of the system.

[0048] The present invention will be described in detail below with reference to various embodiments.

[0049] Example 1

[0050] According to an embodiment of the present invention, an embodiment of a streaming task operation method based on a distributed batch scenario is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] The embodiment of the streaming task operation method based on a distributed batch scenario provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a streaming task operation method based on a distributed batch scenario is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1 The computer system includes a processor 102 (shown as 102a, 102b, ..., 102n) (the processor 102 may include but is not limited to a microcontroller unit (MCU) or a field programmable gate array (FPGA)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the computer system may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS), a network interface, a power supply, and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0052] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0053] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the streaming task operation method based on the distributed batch scenario in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned streaming task operation method based on the distributed batch scenario. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0054] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0055] The display may be, for example, a touch screen liquid crystal display (LCD), which enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0056] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a streaming task operation method based on a distributed batch scenario is provided, which is applied to executors distributedly deployed in multiple campus centers, and all executors share the global streaming read thread number through the target cache.

[0057] Under the above operating environment, this application provides Figure 2 The streaming task operation method based on the distributed batch scenario is shown. Figure 2 This is a flowchart of an optional streaming task operation method based on a distributed batch scenario according to an embodiment of the present invention. Figure 2 As shown, the method includes the following steps:

[0058] Optionally, before receiving the task scheduling instruction distributed by the scheduler, it includes: using a global counter to count the number of threads currently executing the task to obtain the global number of streaming reading threads, wherein, during the counting process, a preset distributed atomic operation command is used to perform statistical conflict avoidance operations on the modification of the same data in the database, and the conflict avoidance operation means that there is only one modification operation for the modification of the same data at the same time point; based on the global streaming reading thread number obtained by statistics, the thread occupancy status is updated.

[0059] In this embodiment, in order to avoid conflicts that may arise when counting the number of threads currently executing tasks in a distributed environment, a global counter is used in combination with a preset distributed atomic operation command to count the number of threads. The global counter (e.g., stream_read_threads:{database_id}) is stored in a distributed cache, such as a Redis cluster, and is used to count the total number of threads performing streaming read operations in real time. It should be noted that the update operation of the counter is implemented in this embodiment through atomic operation commands, such as INCR (increment) or DECR (decrement).

[0060] Atomic operations, in a distributed system, ensure that an operation is either completed in full or not executed in part. This ensures that even when multiple executor nodes attempt to modify the thread count simultaneously, data corruption and concurrency conflicts will not occur. Through this mechanism, this embodiment accurately updates the database InnoDB thread pool occupancy status in real time, ensuring that the concurrency of streaming read jobs remains within a safe range and preventing excessive contention for thread resources from impacting database service stability.

[0061] Optionally, before receiving the task scheduling instruction distributed by the scheduler, it also includes: the scheduler obtains the executor node status through the distributed application coordination service center; the scheduler divides the task into shards based on the job definition and task metadata in the database, and generates a task scheduling instruction, wherein the task metadata includes at least one of the following: the shard to which the task belongs, the task priority, and the task timeout duration; the scheduler distributes the task scheduling instruction to the executors deployed in the multi-campus centers through the message queue.

[0062] In this embodiment, after receiving the job definition and task metadata, the scheduler first obtains the current status of the executor nodes deployed in the multi-campus center through the distributed application coordination service center (such as the ZooKeeper registration center), ensuring that the scheduler can grasp the availability and operation status of all executor nodes, so as to make more reasonable task scheduling decisions. The scheduler then divides the task into shards based on the obtained executor node status, as well as the job definition and task metadata in the database. The purpose of sharding is to improve the parallel processing capability of data reading. By splitting large amounts of data into multiple smaller shards, they can be executed in parallel on multiple executor nodes, significantly improving the speed and efficiency of batch data processing.

[0063] After task division is complete, the scheduler generates specific task scheduling instructions, which include key information such as the shard to which the task belongs, the task priority, and the task timeout duration. Task priority is used to prioritize important or urgent tasks when resources are limited; the task timeout duration is used to set a maximum execution time for tasks, preventing long-running tasks from consuming excessive resources and impacting the scheduling of other tasks. The scheduler distributes these task scheduling instructions to all deployed executor nodes via a message queue (such as Kafka), achieving efficient and orderly task distribution.

[0064] Optionally, the step of the scheduler distributing the task scheduling instructions to the executors deployed in the multi-campus centers through the message queue includes: the scheduler calculates the shard hash value based on the shard to which the task belongs; the scheduler distributes the task scheduling instructions to the executors deployed in the multi-campus centers through the message queue based on the shard hash value and the task priority.

[0065] In this embodiment, when the scheduler is ready to distribute task scheduling instructions to the executor nodes deployed in the multi-campus center, it will first calculate the hash value of the shard to which each task belongs. During the calculation, based on the relevant attributes of the task (such as the shard identifier), the different shards can be mapped to a certain numerical range through a hash function to obtain the shard hash value. The calculation of the shard hash value ensures that each shard can be uniquely identified and can be assigned to a specific executor node in a predictable manner. After the scheduler has the information of the shard hash value and the task priority, it begins to distribute the task scheduling instructions to the executors deployed in the multi-campus center through the message queue (such as the distributed publish-subscribe message platform Kafka).

[0066] Specifically, during the distribution process, the scheduler can decide which executor node to route a specific task scheduling instruction to based on the calculated shard hash value. Each executor node may be assigned one or more hash value intervals to ensure that tasks can be evenly distributed on different nodes, avoiding hot spot issues in task scheduling and improving the system's load balancing capabilities. In addition to the shard hash value, task priority is also an important factor in determining the order of scheduling instructions. When resources are tight, the scheduler will give priority to tasks with higher priorities to ensure that critical tasks can be executed in a timely manner. By inserting high-priority tasks at the front of the queue, this embodiment can respond quickly when resources are available, reducing the waiting time for critical tasks.

[0067] During the distribution process, the present invention relies on the characteristics of the message queue and the flexibility of the scheduling algorithm. For example, the Kafka queue supports multiple partitions, each partition can be regarded as an independent queue, and the scheduler can push the task scheduling instructions to the corresponding partition according to the shard hash value to achieve targeted distribution of tasks.

[0068] In order to further optimize the distribution efficiency of task scheduling instructions and the resource utilization of executor nodes, this embodiment can also dynamically adjust the message queue partitions, such as dynamically adjusting the number and capacity of Kafka partitions according to the resource usage of the executor nodes, to ensure that task scheduling instructions can be quickly and evenly distributed to each node, while reducing the maintenance cost of the message queue. In addition to basic hash value routing, more complex load balancing algorithms such as polling, minimum number of connections, etc. can also be introduced to more finely control the distribution of task scheduling instructions and improve the overall processing capacity and stability of the system. Alternatively, in order to cope with the failure of the executor node, fault tolerance and redundancy mechanisms can be designed in the message queue, such as duplicate message queues, dead letter queues, etc., to ensure that even if some nodes are unavailable, task scheduling instructions can still be correctly distributed and executed.

[0069] Step S201: after receiving the task scheduling instruction distributed by the scheduler, query the number of global streaming reading threads in the distributed cache.

[0070] In this embodiment, when the executor receives a task scheduling instruction from the scheduler, its first action is to query the global streaming read thread count in the distributed cache to determine the number of threads currently performing streaming read operations and whether current system resources are sufficient to support the upcoming streaming read task. The global streaming read thread counter (example key: stream_read_threads:{database_id}) in the distributed cache (e.g., Redis) is continuously updated using pre-configured atomic operations (INCR, DECR) to reflect the current status of streaming read operations in the database thread pool.

[0071] Optionally, after querying the global number of streaming read threads in the distributed cache, it also includes: when the global number of streaming read threads exceeds a preset thread number threshold, adding the task scheduling instruction to the streaming read task waiting queue, wherein the queue length of the streaming read task waiting queue is dynamically adjusted according to the remaining thread resources of the database.

[0072] After querying and obtaining the global number of streaming read threads, this embodiment can also determine whether the current task scheduling instruction is to be executed immediately or delayed based on the comparison between the global number of streaming read threads and the preset thread number threshold. If the queried global number of streaming read threads has exceeded the preset thread number threshold (for example, 70% of the InnoDB thread pool usage), this means that the current system resources are in a tight state. If more streaming read jobs are added, it may cause excessive competition for resources, affecting the stability and response speed of the database service. Therefore, in this case, the executor will not execute the streaming read task immediately, but will add the task scheduling instruction to the streaming read task waiting queue. The dynamic adjustment strategy of the queue length takes into account system resource constraints, the remaining capacity of the thread pool (such as a multiple of the remaining capacity), and the reserved space for high-priority tasks (for example, reserving 20% ​​of the queue capacity for high-priority tasks).

[0073] Furthermore, this embodiment can also monitor the status changes of the thread pool in real time. Once the release of thread resources is detected, for example, when an ongoing streaming read operation is completed and the occupied thread resources are released, the executor will wake up and schedule new task slices from the streaming read task waiting queue in priority order and start executing the streaming read operation, thereby realizing dynamic perception of system resources and efficient scheduling of tasks.

[0074] Step S202 : When the number of global streaming reading threads does not exceed a preset thread number threshold, the job task in the task scheduling instruction is added to the streaming reading task execution queue.

[0075] In this embodiment, when the scheduler generates a task scheduling instruction (including the job's sharding information, priority, and timeout), it first evaluates whether the instruction can be immediately executed under the current environment. This evaluation primarily determines whether the number of global streaming read threads exceeds a preset thread threshold. The preset thread threshold is a value dynamically set based on the database thread pool size and current thread utilization. For example, when the database thread pool utilization reaches 70%, this is considered a critical point, exceeding which may affect the normal service of the database.

[0076] If the global number of streaming read threads does not exceed the preset thread count threshold, meaning the system has sufficient resources to support streaming read operations, the job task will be added to the streaming read task execution queue, awaiting resource allocation and execution. Optionally, the streaming read task execution queue of this embodiment is implemented based on a distributed cache, capable of dynamically adjusting its capacity to accommodate the remaining resources in the current database thread pool. Furthermore, the queue supports priority scheduling, allowing high-priority tasks to be scheduled for execution first, thereby ensuring rapid response to critical business needs.

[0077] Step S203 : When it is detected that the target cache triggers a resource release event, the task to be executed is scheduled from the streaming read task execution queue, and the target thread corresponding to the task to be executed is confirmed.

[0078] In this embodiment, the executor node monitors the target cache (i.e., the Redis instance used to store the number of streaming read threads) to detect the release of database thread pool resources in real time. When the target cache triggers a resource release event, indicating that at least one ongoing streaming read task has completed, thereby freeing up at least one thread resource, the executor node immediately schedules one or more pending tasks from the streaming read task execution queue. The specific number of tasks scheduled depends on the number of freed thread resources and the priority of the tasks in the queue.

[0079] Before scheduling a task to be executed, the executor confirms whether the target thread corresponding to the task is available. During this process, it can first check the current status of the thread resources to ensure that the resource requirements required for task execution are met. If the thread resources are available, the executor will take the task out of the queue and assign the corresponding thread to execute it.

[0080] When executing pending tasks, this embodiment reads the required data from the database using streaming read mode. Streaming read mode is a highly efficient data reading method that can significantly reduce the memory burden on the database and client, particularly when processing large datasets. In streaming read mode, the database does not load all data into the client's memory at once. Instead, it transfers data in batches according to preset data block sizes. Once the client has processed a batch of data, it requests the database to transfer the next batch, and this cycle continues until all data has been fully processed.

[0081] Step S204: Read task data from the database in a streaming read mode according to the task to be executed.

[0082] By maintaining continuous communication between the database and the client, data transmission can be dynamically adjusted according to the client's processing capabilities, avoiding "Out of Memory" (OOM) errors caused by loading large amounts of data at one time. Especially in a distributed environment, the security and efficiency of data processing are significantly improved.

[0083] It's important to note that this embodiment also provides a task retry mechanism. When timed-out tasks fail to complete within the specified timeframe or encounter an exception during execution, the executor forcibly transfers these tasks to a dead-letter queue. Task retry strategies, such as exponential backoff, are then used to reschedule these tasks to address potential system fluctuations or temporary resource shortages. Furthermore, by monitoring queue lengths and wait times in real time, the executor can coordinate resource adjustments to ensure efficient task scheduling and full resource utilization.

[0084] Optionally, according to the task to be executed, the step of reading task data from the database through the streaming reading mode includes: counting the execution time of the task to be executed; when the execution time exceeds the preset execution time threshold, confirming that the task execution is abnormal, and forcibly releasing the target thread through a scheduled job to prevent deadlock.

[0085] In this embodiment, to ensure the normal execution of streaming read tasks and prevent deadlocks caused by prolonged resource occupancy, a task execution duration monitoring and exception handling mechanism is introduced. This embodiment sets a preset execution duration threshold, such as 30 minutes, as the standard for determining whether a task has an execution anomaly. If the actual execution duration of a task exceeds this threshold, the task is considered to have an anomaly, which may be caused by factors such as excessive data volume, inefficient query statements, or insufficient client processing capabilities.

[0086] When a task execution anomaly is confirmed, this embodiment forcibly releases the target thread through a scheduled job. This action is intended to relieve long-term thread occupancy and prevent it from becoming a "zombie thread," which could impact overall database performance and the scheduling of other tasks. This forced thread release mechanism is implemented through a preconfigured scheduled job. When it detects that a thread's occupancy time exceeds a preset threshold, the scheduled job intervenes, interrupts the execution of the anomalous task, and releases the associated thread resources.

[0087] In addition, this embodiment can dynamically adjust the number of concurrent tasks and thread pool size in the streaming read mode according to the system load and task execution status, ensuring efficient use of resources while avoiding service interruptions caused by resource exhaustion.

[0088] Step S205: After completing the task processing, the thread resources corresponding to the target thread are released, and the thread occupancy status in the target cache is updated.

[0089] In this embodiment, when an executor node completes processing of a specific streaming read task, the executor needs to execute an operation to release thread resources corresponding to the target thread and simultaneously update the thread occupancy status stored in the distributed cache (such as a Redis cluster).

[0090] Optionally, after completing task processing, the thread resources corresponding to the target thread are released, and the thread occupancy status in the target cache is updated, including: when releasing the thread resources corresponding to the target thread, obtaining other tasks belonging to the same shard as the task to be executed; batch pulling the thread resources released by the tasks to be executed and other tasks belonging to the same shard, and updating the thread occupancy status in the target cache.

[0091] After the task is completed, the executor immediately releases the target thread previously used for streaming read operations, returning the released resources to the InnoDB thread pool so that other tasks can use them. At the same time, the executor uses pre-defined distributed atomic operation commands (such as DECR) to reduce the number of current streaming read threads recorded in the target cache, ensuring the accuracy of thread occupancy status and providing the latest resource usage information to other executor nodes or schedulers.

[0092] In order to further optimize the scheduling and utilization efficiency of resources, this embodiment adopts an additional strategy: when releasing the resources of a single streaming read task, the executor will check whether there are tasks to be executed that belong to the same shard as other processed tasks. If so, the executor will pull these tasks in batches and start processing immediately using the resources just released, thereby reducing the waiting time of tasks and the idle period of resources, and improving the concurrent processing capabilities of streaming read tasks. The implementation of this strategy requires the executor to have the ability to identify tasks and manage resources, that is, the executor needs to be able to identify which tasks belong to the same shard, which is usually based on the shard identification information in the task metadata. After confirming the available resources, the executor needs to be able to immediately allocate these resources to the corresponding tasks to be executed to avoid unnecessary waiting and waste of resources.

[0093] To ensure the real-time and accurate status of thread occupancy, the executor must utilize the distributed cache's atomic operations when releasing thread resources and updating the target cache state to avoid data conflicts during concurrent updates. Furthermore, the executor should monitor thread resource change events (e.g., through Redis's PUB / SUB mechanism). Once a resource state is updated, it broadcasts the event to other nodes, facilitating rapid consistency across the entire system and supporting more efficient resource awareness and task scheduling.

[0094] Through the above steps, after receiving the task scheduling instruction distributed by the scheduler, the global streaming read thread number in the distributed cache can be queried; when the global streaming read thread number does not exceed the preset thread number threshold, the job task in the task scheduling instruction is added to the streaming read task execution queue; when the target cache triggers a resource release event, the task to be executed is scheduled from the streaming read task execution queue, and the target thread corresponding to the task to be executed is confirmed; according to the task to be executed, the task data is read from the database through the streaming read mode; after completing the task processing, the thread resources corresponding to the target thread are released, and the thread occupancy status in the target cache is updated. In this embodiment, the global streaming read thread number can be compared with the preset thread number threshold, and the comparison result can be used to confirm whether the job task is added to the streaming read task execution queue to avoid excessive consumption of thread resources and affecting the execution of other important tasks. At the same time, after completing the task processing, the thread resources corresponding to the target thread are released in time, and the thread occupancy status in the target cache is updated, so that the thread resources can be kept in a reasonable state, thereby solving the technical problem in the related technology that the streaming read strategy in the distributed batch scenario is prone to long task processing time, excessive consumption of thread resources, and affecting database use.

[0095] The following describes in detail another optional specific implementation.

[0096] In distributed batch scenarios, the sheer volume of data processed poses significant challenges for both database servers and applications. Streaming reads can alleviate memory pressure on both sides, but due to the nature of streaming reads, they can sometimes impact database availability and hinder online transactions. To address this, embodiments of the present invention provide a reliable method for concurrent streaming reads based on multi-shard jobs in distributed batch scenarios.

[0097] Figure 3 FIG. 1 is a schematic diagram of thread scheduling during the interaction between each application container and the database in an optional distributed batch scenario according to an embodiment of the present invention. Figure 3 As shown, the database includes: a connection pool (including multiple idle connections and active connections), a thread pool (including thread 1, thread 2...thread X that are executing tasks, executing various waiting task requests / instructions) and a data log.

[0098] To ensure high availability of scheduling, this implementation implements a multi-park distributed deployment of each executor node in the system. This may cause streaming reads from multiple shards to reach the thread pool limit of the database (this implementation is illustrated using the Innodb database, but is not limited to this Innodb database). In daily use, to ensure database availability, a distributed cache is added to the executor side (deploy a distributed cache (such as a Redis cluster) to share the number of streaming read threads, and a global counter (key such as stream_read_threads:{database_id}) is used in combination with atomic operations (INCR / DECR) to update thread occupancy status in real time to ensure concurrency safety.

[0099] Optionally, this implementation method uses Redis master-slave replication to achieve eventual consistency of state between nodes, and uses the PUB / SUB mechanism to push resource change events, trigger task wake-up, avoid thread number statistics conflicts based on Redis atomic operations, monitor InnoDB thread pool state changes (such as thread release), and achieve real-time resource perception; Redis persistence (RDB / AOF) and sentinel failover are used to ensure data reliability, and deadlocks are prevented by forced release of task timeouts (such as 30 minutes) and abnormal node detection and retry of queue tasks. At the same time, by monitoring the thread pool usage threshold (70%), stream reading task sharding optimization, and scheduled job polling resource availability, the number of concurrency is accurately controlled to improve scheduling efficiency while ensuring database availability) to share the number of streaming read threads used by storage applications.

[0100] Figure 4 This is a system architecture diagram for implementing multi-shard job tasks in an optional distributed batch scenario according to an embodiment of the present invention. Figure 4 As shown, it includes: scheduler controller, queue queue, executor executor, cache cache, streaming read quantity streamCount and database database.

[0101] Among them, the scheduler controller is the manager responsible for job scheduling, job execution process control, and resource management in the batch scheduling framework. It is the brain of the batch framework.

[0102] Batch executor: It is the node of each application, used to receive scheduling instructions issued by the batch controller and execute the job program.

[0103] Figure 4The interaction between the modules is as follows: The scheduler obtains the executor node status through the ZooKeeper registry, generates scheduling instructions based on the job definition and metadata in the database, and distributes tasks to executors deployed across multiple campuses via a message queue (such as Kafka). After receiving a task, the executor first queries the global streaming read thread count (for example, innodb_threads usage) in a distributed cache (such as Redis). If it exceeds a set threshold (such as 70%), the task is added to the streaming read task waiting queue (implemented via a Redis list or Kafka topic). The queue capacity is dynamically adjusted based on the remaining database thread resources. When the cache detects the release of resources (through Redis PUB / SUB or timed polling), it triggers a scheduled job to pull tasks with the same shard in batches from the queue according to priority (such as LPOP high-priority tasks, List POP, the LPOP command is used to remove and return an element from the head of the queue (the leftmost element), which is suitable for implementing a first-in-first-out queue structure. The element that is first added to the queue is the first to be processed). The executor reads data from the database in batches through streaming read mode (using MySQL's cursor mechanism), and updates the thread occupancy status in the cache in real time during the process. After the executor completes data processing, it writes the results back to the database and releases thread resources. At the same time, ZooKeeper monitors the health status of the node and triggers task reallocation in case of an abnormality. Control signals (such as scheduling instructions and resource status) are transmitted through the message queue and cache, and the data stream (task metadata, sharding results) are synchronized through the database and cache, ultimately achieving controllable resource competition, efficient task scheduling and data consistency assurance).

[0104] In addition, the system for implementing multi-shard jobs in distributed batch scenarios can also include a batch console, a database, and a ZooKeeper registry. The batch console is provided to operations and maintenance personnel, primarily used to display information such as job scheduling and execution progress, and to facilitate emergency manual adjustments to batch parameters. The database stores metadata for the batch framework and information such as application schedules, job definitions, and job execution status. The ZooKeeper registry is responsible for batch controller and executor registration, batch task message delivery, and parameter configuration.

[0105] On the executor side, a waiting queue for streaming read tasks is maintained. The queue is designed based on dynamic capacity adjustment and priority scheduling to ensure orderly task processing during resource contention. Capacity is set based on dynamic threshold adaptation (e.g., a multiple of the thread pool's remaining capacity) and system resource constraints (a fixed threshold combined with memory limits). Furthermore, 20% of the capacity is reserved for high-priority tasks. The enqueue logic is triggered when the InnoDB thread pool utilization reaches 70%. Task metadata (including shard, priority, and timeout) is stored in a distributed queue, and high-priority tasks are placed at the head of the queue. The dequeue logic monitors resource release events, prioritizes high-priority tasks, and forcibly transfers timed-out tasks to a dead-letter queue. Batch fetching of tasks from the same shard is supported to reduce overhead. To balance queue efficiency and scheduling performance, a dynamic scaling mechanism (adjusting queue capacity based on throughput), shard hash allocation and merging optimization (avoiding queue hotspots), a timeout retry strategy (exponential backoff), and real-time monitoring feedback (linking queue length and wait time to resource adjustments) are implemented.

[0106] By combining persistent storage, distributed locking, and failover (e.g., utilizing Redis Sentinels), this embodiment achieves high throughput, low latency (average task wait time <5 minutes), and reliable scheduling, ensuring stable online database transactions while improving batch processing efficiency. Furthermore, this implementation also employs a series of measures to ensure controllable streaming read concurrency without impacting database scheduling performance. By monitoring the usage of multi-sharded applications, when it reaches 70% of the InnoDB thread pool, the executor suspends scheduling and places the task in the streaming read queue, awaiting resources. This effectively prevents application performance degradation or system crashes due to insufficient resources.

[0107] In order to ensure that the application can be woken up in time to schedule the current streaming read shard job when resources are available (when executing multi-shard streaming reads concurrently, the present invention avoids resource competition through distributed cache and dynamic threshold control mechanism: deploy distributed cache (such as Redis cluster) on the executor node, and use global counter to monitor the database InnoDB thread pool usage in real time. When the occupancy rate reaches the 70% threshold, the new task is paused and enters the streaming read task waiting queue. The queue capacity is dynamically adjusted based on the number of remaining threads (such as remaining capacity × 2), and resource allocation is optimized through priority scheduling (high priority tasks are queued). At the same time, shard hash allocation and batch processing strategies are adopted to distribute tasks to different queues according to shard hash. When resources are released, tasks of the same shard are pulled in batches (such as 10 at a time) to reduce network overhead and thread contention), a timed job is added to the executor. This timed job will periodically poll to check whether resources are available. Once it finds that resources are available, it will wake up the application and schedule the current streaming read shard job (generally data distribution). In this way, the number of concurrent streaming reads can be ensured to be within a controllable range, avoiding impact on database scheduling performance.

[0108] Furthermore, this embodiment also adds some other measures to optimize the concurrent performance of streaming reading. For example, the streaming reading tasks are reasonably sharded to reduce the resource consumption of a single task. At the same time, the efficiency of streaming reading is improved by adjusting database parameters, optimizing query statements, etc.

[0109] Through the above implementation, a series of measures have been implemented on the executor side to ensure controllable streaming read concurrency without affecting database scheduling performance. These measures include maintaining a queue for streaming read tasks, adding scheduled jobs, and optimizing streaming read concurrency. These measures enable better control of streaming read concurrency and avoid negative impacts on database scheduling performance.

[0110] The following describes it in detail with reference to another embodiment.

[0111] Example 2

[0112] A streaming task operation device based on a distributed batch scenario provided in this embodiment includes multiple implementation units, each implementation unit corresponds to each implementation step in the above-mentioned embodiment 1. Its specific implementation method and beneficial effects can be referred to the above-mentioned method embodiment and will not be repeated here.

[0113] According to another aspect of an embodiment of the present invention, a streaming task operation device based on a distributed batch scenario is also provided, which is applied to executors distributedly deployed in multiple campus centers, and all executors share the global streaming read thread number through the target cache.

[0114] Figure 5is a schematic diagram of an optional streaming task operation device based on a distributed batch scenario according to an embodiment of the present invention, such as Figure 5 As shown, the streaming task operation device based on the distributed batch scenario may include: a thread number query unit 51, a task queue adding unit 52, a task scheduling unit 53, a task execution unit 54, and a thread state updating unit 55.

[0115] The thread number query unit 51 is used to query the number of global streaming reading threads in the distributed cache after receiving the task scheduling instruction distributed by the scheduler.

[0116] The task queue adding unit 52 is configured to add the job task in the task scheduling instruction to the streaming reading task execution queue when the number of global streaming reading threads does not exceed a preset thread number threshold.

[0117] The task scheduling unit 53 is used to schedule the task to be executed from the streaming read task execution queue when a resource release event is detected that the target cache triggers the resource release event, and to confirm the target thread corresponding to the task to be executed;

[0118] The task execution unit 54 is configured to read task data from the database in a streaming reading mode according to the task to be executed.

[0119] The thread status updating unit 55 is used to release the thread resources corresponding to the target thread after completing the task processing, and update the thread occupancy status in the target cache.

[0120] The above-mentioned streaming task job device based on the distributed batch scenario can query the global streaming reading thread number in the distributed cache through the thread number query unit 51 after receiving the task scheduling instruction distributed by the scheduler, and add the job task in the task scheduling instruction to the streaming reading task execution queue through the task queue adding unit 52 when the global streaming reading thread number does not exceed the preset thread number threshold. When the task scheduling unit 53 monitors the target cache triggering resource release event, it schedules the task to be executed from the streaming reading task execution queue and confirms the target thread corresponding to the task to be executed. The task execution unit 54 reads the task data from the database through the streaming reading mode according to the task to be executed. After completing the task processing, the thread resources corresponding to the target thread are released through the thread status update unit 55, and the thread occupancy status in the target cache is updated. In this embodiment, the global number of streaming read threads can be compared with the preset thread number threshold, and the comparison result can be used to confirm whether the job task is added to the streaming read task execution queue to avoid excessive consumption of thread resources and affect the execution of other important tasks. At the same time, after completing the task processing, the thread resources corresponding to the target thread are released in time, and the thread occupancy status in the target cache is updated, so that the thread resources can be maintained in a reasonable state, thereby solving the technical problems in related technologies that the use of streaming read strategies in distributed batch scenarios is prone to long task processing time, excessive consumption of thread resources, and impact on database usage.

[0121] Optionally, the streaming task operation device based on a distributed batch scenario also includes: a counting unit, which is used to use a global counter to count the number of threads currently executing the task before receiving the task scheduling instruction distributed by the scheduler, and obtain the global number of streaming reading threads, wherein, during the counting process, a preset distributed atomic operation command is used to perform statistical conflict avoidance operations on the modifications of the same data in the database, and the conflict avoidance operation means that there is only one modification operation for the modification of the same data at the same time point; a thread occupancy status update unit, which is used to update the thread occupancy status based on the global streaming reading thread number obtained by statistics.

[0122] Optionally, the streaming task operation device based on a distributed batch scenario also includes: a node status acquisition unit, which is used by the scheduler to obtain the executor node status through the distributed application coordination service center before receiving the task scheduling instruction distributed by the scheduler; a slicing unit, which is used by the scheduler to divide the task into slicings based on the job definition and task metadata in the database, and generate task scheduling instructions, wherein the task metadata includes at least one of the following: the slicing to which the task belongs, the task priority, and the task timeout duration; an instruction distribution unit, which is used by the scheduler to distribute the task scheduling instructions to the executors deployed in the multi-campus centers through the message queue.

[0123] Optionally, the instruction distribution unit includes: a shard hash value calculation module, which is used by the scheduler to calculate the shard hash value based on the shard to which the task belongs; an instruction distribution module, which is used by the scheduler to distribute the task scheduling instructions to the executors deployed in the multi-park centers through the message queue based on the shard hash value and task priority.

[0124] Optionally, the task execution unit includes: a task execution time statistics module, used to count the execution time of the task to be executed; a task exception handling module, used to confirm the task execution exception when the execution time exceeds the preset execution time threshold, and forcibly release the target thread through a scheduled job to prevent deadlock.

[0125] Optionally, the thread status update unit includes: other same-slice task acquisition module, used to acquire other tasks belonging to the same slice as the task to be executed when releasing the thread resources corresponding to the target thread; a thread resource batch pulling module, used to batch pull the thread resources released by the tasks to be executed and other tasks belonging to the same slice, and update the thread occupancy status in the target cache.

[0126] Optionally, the streaming task operation device based on the distributed batch scenario also includes: a waiting queue joining unit, which is used to add the task scheduling instruction to the streaming reading task waiting queue after querying the global streaming reading thread number in the distributed cache, when the global streaming reading thread number exceeds the preset thread number threshold, wherein the queue length of the streaming reading task waiting queue is dynamically adjusted according to the remaining thread resources of the database.

[0127] The above-mentioned streaming task operation device based on a distributed batch scenario can also include a processor and a memory. The above-mentioned thread number query unit 51, task queue joining unit 52, task scheduling unit 53, task execution unit 54, thread status update unit 55, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0128] The processors mentioned above include a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and kernel parameters can be adjusted to enable concurrent streaming jobs based on multi-shard jobs in distributed batch scenarios.

[0129] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0130] Example 3

[0131] An embodiment of the present application may provide an electronic device, Figure 6This is a structural block diagram of an electronic device for executing a streaming task operation method based on a distributed batch scenario according to an embodiment of the present application. Figure 6 As shown, the electronic device may include: one or more ( Figure 6 Only one is shown) processor 602, memory 604, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0132] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the streaming task operation method and device based on distributed batch scenarios in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned streaming task operation method based on distributed batch scenarios. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0133] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: after receiving the task scheduling instruction distributed by the scheduler, query the global streaming read thread number in the distributed cache; when the global streaming read thread number does not exceed the preset thread number threshold, add the job task in the task scheduling instruction to the streaming read task execution queue; when monitoring the target cache triggering resource release event, schedule the task to be executed from the streaming read task execution queue, and confirm the target thread corresponding to the task to be executed; according to the task to be executed, read the task data from the database through the streaming read mode; after completing the task processing, release the thread resources corresponding to the target thread, and update the thread occupancy status in the target cache.

[0134] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), or a PAD. Figure 6 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 6 Different configurations shown.

[0135] Those skilled in the art will understand that all or part of the steps in the various streaming task operation methods based on distributed batch scenarios in the above-mentioned embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0136] Example 4

[0137] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the distributed batch scenario-based streaming task operation method provided in the first embodiment.

[0138] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the streaming task operation methods based on a distributed batch scenario in the above-mentioned embodiment 1.

[0139] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0140] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the streaming task operation method based on a distributed batch scenario described in various embodiments of the present application.

[0141] The present application also provides a computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the streaming task operation method based on the distributed batch scenario described in each embodiment of the present application.

[0142] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0143] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0145] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0146] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0148] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A streaming task operation method based on a distributed batch scenario, characterized in that: Applicable to executors deployed in a multi-campus center, all of which share the global streaming read thread count through the target cache, including: After receiving the task scheduling instruction issued by the scheduler, query the number of global streaming reading threads in the distributed cache; When the number of global streaming read threads does not exceed a preset thread number threshold, adding the job task in the task scheduling instruction to the streaming read task execution queue; In the case of monitoring that a target cache triggers a resource release event, scheduling a task to be executed from the streaming read task execution queue, and confirming a target thread corresponding to the task to be executed; According to the task to be executed, task data is read from the database in a streaming read mode; After completing the task processing, the thread resources corresponding to the target thread are released, and the thread occupancy status in the target cache is updated.

2. The streaming task operation method according to claim 1, characterized in that: Before receiving the task scheduling instruction issued by the scheduler, it also includes: A global counter is used to count the number of threads currently executing tasks to obtain the global number of streaming reading threads. During the counting process, a preset distributed atomic operation command is used to perform statistical conflict avoidance operations on modifications to the same data in the database. The conflict avoidance operation means that there is only one modification operation at the same time point for the modification of the same data. Based on the statistically obtained number of global streaming reading threads, the thread occupancy status is updated.

3. The streaming task operation method according to claim 1, characterized in that: Before receiving the task scheduling instruction issued by the scheduler, it also includes: The scheduler obtains the executor node status through the distributed application coordination service center; The scheduler divides the task into slices based on the job definition and task metadata in the database and generates the task scheduling instruction, wherein the task metadata includes at least one of the following: the slice to which the task belongs, the task priority, and the task timeout duration; The scheduler distributes the task scheduling instructions to the executors deployed in the multi-campus centers through the message queue.

4. The streaming task operation method according to claim 3, characterized in that: The step of the scheduler distributing the task scheduling instruction to the executors deployed in the multi-campus center through the message queue includes: The scheduler calculates a shard hash value based on the shard to which the task belongs; The scheduler distributes the task scheduling instructions to the executors deployed in the multi-campus center through a message queue based on the shard hash value and the task priority.

5. The streaming task operation method according to claim 1, characterized in that: According to the task to be executed, the step of reading task data from the database in a streaming read mode includes: Counting the execution time of the tasks to be executed; When the execution time exceeds a preset execution time threshold, the task execution is confirmed to be abnormal, and the target thread is forcibly released through a scheduled job to prevent deadlock.

6. The streaming task operation method according to claim 1, characterized in that: After completing task processing, releasing thread resources corresponding to the target thread and updating the thread occupancy status in the target cache include: When releasing the thread resources corresponding to the target thread, obtaining other tasks that belong to the same slice as the task to be executed; Pulling the thread resources released by the tasks to be executed and other tasks belonging to the same slice in batches, and updating the thread occupancy status in the target cache.

7. The streaming task operation method according to claim 1, characterized in that: After querying the number of global streaming reader threads in the distributed cache, it also includes: When the number of global streaming read threads exceeds a preset thread number threshold, the task scheduling instruction is added to a streaming read task waiting queue, wherein the queue length of the streaming read task waiting queue is dynamically adjusted according to the remaining thread resources of the database.

8. A streaming task operation device based on a distributed batch scenario, characterized in that: Applicable to executors deployed in a multi-campus center, all of which share the global streaming read thread count through the target cache, including: A thread number query unit is used to query the number of global streaming reading threads in the distributed cache after receiving a task scheduling instruction issued by the scheduler; A task queue adding unit, configured to add the job task in the task scheduling instruction to the streaming reading task execution queue when the number of the global streaming reading threads does not exceed a preset thread number threshold; A task scheduling unit is used to schedule a task to be executed from the streaming read task execution queue and confirm a target thread corresponding to the task to be executed when a target cache triggers a resource release event; A task execution unit, configured to read task data from a database in a streaming reading mode according to the task to be executed; The thread status updating unit is used to release the thread resources corresponding to the target thread after completing the task processing, and update the thread occupancy status in the target cache.

9. An electronic device, characterized in that: It includes one or more processors and a memory, the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the streaming task operation method based on a distributed batch scenario as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the streaming task operation method based on a distributed batch scenario described in any one of claims 1 to 7 are implemented.

Images