A task processing method, system and related equipment

Abstract

This invention discloses a task processing method, system, and related equipment, belonging to the field of distributed computing technology. The method is applied to a distributed system containing multiple nodes and includes: encapsulating a task to be processed into a first message and publishing it to a unified message queue of the system; subscribing to and obtaining the first message from the unified message queue through a business node, and injecting the obtained first message into a local message queue within the business node; and obtaining a second message from the local message queue through multiple pre-set workers within the business node, and executing the task corresponding to the second message.

Description

Technical Field

[0001] This invention relates to the field of distributed computing technology, specifically to a task processing method, system, and related equipment. Background Technology

[0002] Parallel computing refers to the process of using multiple computing resources simultaneously to solve computational problems. It is an effective means of improving the computing speed and processing power of computer systems. Its basic idea is to use multiple processors to collaboratively solve the same problem, that is, to decompose the problem into several parts, each of which is computed in parallel by an independent processor. Traditional parallel computing models include the Parallel Random Access Machine (PRAM) model and the Bulk Synchronous Parallel Computing (BSP) model.

[0003] PRAM is a shared-memory model within the Multiple Instruction Multiple Data (MIMD) parallel machine paradigm. Its principle assumes an infinitely large shared memory and multiple processors with identical functionality, allowing each processor to access the shared memory at any given time. BSP, on the other hand, resembles a parallel random access model. The BSP model abstracts parallel computing into H modules: a set of processors, a global communication network for sending messages, and a roadblock synchronization mechanism between processors. The basic execution unit for parallel computing in the BSP model is the superstep.

[0004] However, PRAM relies on an idealized shared memory architecture, which suffers from insufficient robustness in practical applications due to issues such as network latency and data synchronization. BSP, on the other hand, strictly constrains the computation process, leading to difficulties in business decoupling and high implementation complexity. Therefore, a method is urgently needed to address the aforementioned problems in related technologies. Summary of the Invention

[0005] This application provides a task processing method, system, and related equipment to solve the problems of implementation difficulties and poor robustness caused by traditional parallel computing models (such as PRAM and BSP) due to their overly idealized nature and high coupling with business logic.

[0006] This application provides a task processing method, which is applied to a distributed system containing multiple nodes, including: The task to be processed is encapsulated into a first message and published to the unified message queue of the system; The first message is subscribed to and retrieved from the unified message queue through the business node, and the retrieved first message is injected into the local message queue inside the business node. The second message is retrieved from the local message queue by multiple pre-set workers within the business node, and the task corresponding to the second message is executed.

[0007] This application embodiment also provides a distributed system, the system comprising: A unified message queue is used to transmit the first message, which is encapsulated by the task to be processed. At least one business node, including: The message interface unit is used to obtain the first message from the unified message queue and inject it into the local message queue; A local message queue is used to buffer the first message from the message interface unit; Multiple workers are used to retrieve a second message from the local message queue and execute the task corresponding to the second message.

[0008] This application embodiment also provides a task processing device, including: The message publishing module is used to encapsulate the tasks to be processed into a first message and publish it to the unified message queue of the system. The message injection module is used to subscribe to and obtain the first message from the unified message queue through the business node, and to inject the obtained first message into the local message queue inside the business node. The task execution module is used to obtain a second message from the local message queue through multiple workers pre-set within the business node, and execute the task corresponding to the second message.

[0009] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. The processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform steps in the task processing method provided in this application.

[0010] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps in the task processing method provided in this application.

[0011] This application also provides a computer program product that stores instructions that, when executed by a computer, cause the computer to perform the steps in the task processing method provided in this application.

[0012] The task processing method provided in this application encapsulates the task to be processed into a first message and distributes it across nodes via a unified message queue. Then, business nodes inject the first message into their local message queues. Finally, multiple workers within the business nodes concurrently retrieve and execute the corresponding second message, constructing a two-level message queue distributed parallel processing architecture. This architecture effectively solves the implementation difficulties and insufficient robustness of the traditional PRAM model due to its reliance on idealized shared memory. It achieves reliable inter-node collaboration through message communication rather than memory sharing. Simultaneously, by decoupling the business message (the second message) from the unified message format, it overcomes the shortcomings of the BSP model, such as excessive coupling with business logic and poor versatility. This enables the parallel computing system to possess both high reliability and scalability, while also flexibly adapting to diverse business processing needs, achieving a balance between high performance and versatility. Attached Figure Description

[0013] Figure 1 A flowchart illustrating a task processing method provided for an exemplary embodiment of this application; Figure 2 A schematic diagram of the basic system architecture of the message queue-based two-level concurrency mechanism in the task processing method provided for an exemplary embodiment of this application; Figure 3 A schematic diagram of the internal architecture of a business node (task node) in a task processing method provided for an exemplary embodiment of this application; Figure 4 A schematic diagram illustrating task splitting of multi-level tasks in a task processing method provided as an exemplary embodiment of this application; Figure 5 A schematic diagram of the structure of a distributed system provided for an exemplary embodiment of this application; Figure 6 A schematic diagram of the structure of a task processing apparatus provided for an exemplary embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0014] The principles and spirit of the invention will now be described with reference to several exemplary embodiments shown in the accompanying drawings. These embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way.

[0015] The following is a description of the terms used in this application: A unified message queue is a centralized message passing component used in a distributed system for asynchronous, decoupled communication between all nodes. It employs standardized interfaces and protocols to carry and route messages encapsulating task information. It serves as the core communication infrastructure connecting gateway nodes, authentication nodes, and business nodes, responsible for task distribution and result feedback across nodes. Its design supports various communication modes, including master-slave architecture, point-to-point, and publish-subscribe.

[0016] A local message queue refers to a message buffering and scheduling component deployed within a business node. It receives task messages from a unified message queue and temporarily queues them in the node's memory, serving as a task pool for multiple workers within the node to concurrently acquire tasks. Its core function is to achieve task load balancing, traffic shaping, and concurrency control within the node, avoiding resource contention among workers and task processing congestion.

[0017] The first message refers to the message entity transmitted in a unified message queue. It adopts a general structured format design, and its payload encapsulates second message data related to specific business operations. The first message typically contains metadata fields such as message type, sequence number, category identifier, direction identifier, and binary payload. It is mainly used for system-level message routing, sequence management, transmission guarantee, and security control, and its format is decoupled from specific business logic.

[0018] The second message refers to the message entity that carries the semantics of specific business operations. It is stored in the payload field of the first message and is parsed and executed by workers within the business node. The structure of the second message is dynamically defined according to the type of business operation, including various types such as data caching, data modification, data query, and calculation tasks. Its design goal is to abstract business logic into standardized messages and achieve separation of task processing and communication architecture.

[0019] An atomic subtask is the smallest indivisible unit of task in parallel computing. The execution of this unit satisfies the atomicity requirement: all its operations either complete successfully or are not executed at all, and it is scheduled, transmitted, and processed as a whole in a distributed environment. Atomicious subtasks are formed by decomposing and recombining complex tasks, and they are the fundamental unit for ensuring consistency and reliability in distributed processing.

[0020] A business node is a functional node in a distributed system that specifically performs data processing or computation tasks. Internally, it includes a local message queue, a worker pool, and a task scheduling mechanism. It is responsible for receiving tasks from a unified message queue, buffering and distributing tasks within the node, and organizing workers to execute tasks concurrently. Business nodes are the actual providers of parallel computing capabilities and can dynamically expand or shrink according to system load.

[0021] A worker is the smallest processing unit that concurrently executes tasks within a business node, typically implemented as a thread, coroutine, or process. Workers continuously listen to the local message queue, retrieve messages, and execute corresponding business operations. Upon completion, they can return results or retrieve new tasks. Multiple workers run in parallel, collectively constituting the processing capacity of a business node; their number can be configured based on node resources.

[0022] A gateway node is a functional node responsible for managing message routing and forwarding within the system. It undertakes responsibilities such as fan-in / fan-out of a unified message queue, routing, protocol conversion, and traffic control. It supports flexible interconnection between different nodes (such as M:N communication topologies) and provides a unified network access point, making it a key infrastructure component ensuring efficient and reliable message delivery.

[0023] An authentication node is a functional node responsible for managing node identities and access control in a distributed system. It provides node registration, authentication, authorization, and session management services, ensuring that only legitimate nodes can access the unified message queue and publish or subscribe to tasks. Authentication nodes are fundamental to system security, preventing unauthorized access and malicious task injection.

[0024] As described in the background section, parallel computing is a key means to improve the data processing capabilities of computer systems. It decomposes large-scale computing tasks and has multiple processing units work together to complete them. In the field of distributed parallel computing, several models have been proposed. Among them, the PRAM (Parallel Random Access Machine) model is a theoretical model that assumes the existence of an infinitely large shared memory accessible to all processors. While this model simplifies algorithm design, its idealized assumptions of "infinite shared memory" and "precise synchronization" are difficult to achieve in real-world hardware and network environments, resulting in poor system robustness and difficulties in engineering implementation. Another common model is the BSP (Global Synchronous Parallel) model, which divides the computation process into a series of supersteps. Each superstep includes three stages: local computation, global communication, and roadblock synchronization. Although the BSP model has a clear structure, its strict computation-communication-synchronization cycle is highly coupled with diverse business logic, making it difficult to build a flexible and general-purpose business processing framework. The system design is complex and not easily adaptable to different application scenarios.

[0025] Furthermore, existing message queue-based distributed systems typically employ a single-level message passing mechanism. This mechanism often tightly couples communication protocols with business logic, resulting in complex message formats that are difficult to extend and cannot effectively guarantee the integrity and reliability of cross-node tasks. When faced with scenarios requiring the safe and efficient decomposition and parallel execution of complex tasks while ensuring eventual consistency, existing solutions fall short in terms of architectural versatility, system robustness, and business decoupling.

[0026] To address the aforementioned problems in related technologies, embodiments of this application provide a task processing method, system, and related equipment. This method encapsulates the task to be processed into a first message and distributes it across nodes via a unified message queue. Then, business nodes inject the first message into their local message queues. Finally, multiple workers within the business nodes concurrently retrieve and execute the corresponding second message, thus constructing a distributed parallel processing architecture with a two-level message queue. This architecture effectively solves the implementation difficulties and insufficient robustness of the traditional PRAM model due to its reliance on idealized shared memory. It achieves reliable inter-node collaboration through message communication rather than memory sharing. Simultaneously, by decoupling the unified message format from the business message (i.e., the second message), it overcomes the shortcomings of the BSP model, such as excessive coupling with business logic and poor versatility. This enables the parallel computing system to possess both high reliability and scalability, while also flexibly adapting to diverse business processing needs, achieving a balance between high performance and versatility.

[0027] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart illustrating a task processing method provided as an exemplary embodiment of this application. The method is applied to a distributed system comprising multiple nodes, which are nodes with various functions; the business node described below is one type specifically responsible for executing tasks. Figure 1 As shown, the method includes: Step 110: Encapsulate the task to be processed into a first message and publish it to the system's unified message queue.

[0029] For example, the first message refers to a general communication message transmitted in the unified message queue. The first message can also be called a first-level message construct, i.e., the CmdPrimitive message structure. It carries task data through the payload_data field and uses the unified message queue to publish tasks across nodes. The first message is a standardized message format independent of specific business logic, primarily used for cross-node task routing, sequence management, and transmission assurance. Its structure includes metadata such as message type, sequence number, and classification identifier to control the communication process, as well as a payload field to carry the actual business data.

[0030] A unified message queue refers to a centralized communication infrastructure connecting all nodes in a system, and is the core communication hub in this application's embodiments. Its design meets the following requirements to ensure reliable and efficient cross-node communication: 1. It adopts a master-slave server architecture, where the master server can broadcast information to all slave servers.

[0031] 2. Supports point-to-point communication mode, and data transmission adopts the "request-response-confirmation" or "transmission-confirmation" mechanism to improve efficiency while ensuring process traceability.

[0032] 3. Includes authentication nodes (1:N configuration), used for node registration, authentication, and general information publishing.

[0033] 4. Includes gateway nodes (proxy servers) for fan-in / fan-out message routing to support M:N network communication topologies.

[0034] 5. Provide basic communication guarantees for upper-layer load sharing, multi-process or polling mechanisms.

[0035] 6. The data transmission format is clearly defined, using a specific frame structure to decouple the upper-layer business logic from the lower-layer transmission mode.

[0036] The unified message queue is responsible for task distribution and result feedback across nodes, and is the core component for implementing distributed task scheduling.

[0037] In some exemplary embodiments, the first message encapsulates encoded data of the second message.

[0038] The payload_data field of the first message encapsulates the encoded data of the second message, thus decoupling communication from business logic. The second message refers to the business message encapsulated within the payload of the first message, including specific business message types such as SQLREQV1 and TASKREQV1. Its structure is dynamically defined based on the business operation type (such as data query, calculation task, etc.), serving as the carrier of business logic. It is parsed and executed only within the business node by workers, achieving separation of business processing from communication transmission.

[0039] In some exemplary embodiments, the data structure of the first message includes: a message type field, a message sequence number field, a message classification identifier field, and a message payload field; the message payload field is used to carry the encoded data of the second message.

[0040] For example, the first message uses a general structure defined by Protobuf, whose fields may include: MsgType, fixed32 serial_number, MsgID, and bytes payload_data. MsgType corresponds to the message type field, fixed32 serial_number corresponds to the message sequence number field, MsgID corresponds to the message category identifier field, and bytes payload_data corresponds to the message payload field, which carries the encoded data of the second message. This structure uses the classic Type-Value construction, ensuring simplicity while maintaining good scalability through type and msg_id. The serial_number serves as a temporary unique identifier used to match requests and responses.

[0041] In some exemplary embodiments, the task encapsulated by the first message and delivered via a unified message queue is atomic; the method further includes: Before encapsulating the task to be processed into the first message, the task to be processed is decomposed into multiple independent atomic subtasks. For each atomic subtask, the atomic task is encapsulated into multiple corresponding first messages to execute subsequent steps.

[0042] Among them, atomic subtasks refer to the smallest indivisible task unit that meets the atomicity requirement. Their execution usually satisfies the characteristic that all operations are either all completed or none are executed. They are independent processing units obtained by splitting complex tasks and are used to ensure the reliability and consistency of task processing in a distributed environment.

[0043] Specifically, the system first encapsulates the task to be processed into a uniformly formatted CmdPrimitive message (i.e., the first message). The task parameters and instructions are encoded into business messages such as SQLREQV1 or TASKREQV1 (i.e., the second message) and placed in the payload_data field of the first message. This first message is published to a unified message queue, which is then retrieved by the business nodes that have subscribed to the queue. The business nodes parse the first message, extract the second message, and inject it into their local message queue. Finally, the workers within the node execute the messages concurrently. If the task is a multi-level, non-atomic task, it is first split into multiple atomic subtasks, which are then encapsulated into independent first messages for distribution.

[0044] By designing an encoding / decoding mechanism that separates the first and second messages, communication control and business logic are decoupled. This allows the unified message queue to handle only standardized communication messages, while business nodes can focus on parsing and executing diverse business messages. This approach directly addresses the high coupling and poor versatility of traditional BSP models, enabling the system to flexibly support various business types such as information caching, data querying, and computational tasks. Simultaneously, by enforcing task atomicity and implementing a task splitting mechanism, each subtask is ensured to be processed as a complete unit in a distributed environment, avoiding state inconsistencies caused by partial success. This solves the robustness issues of the PRAM model due to idealized synchronization and shared memory, improving system reliability and processing consistency. Ultimately, these methods collectively achieve a high-performance parallel computing architecture that is both highly versatile and guarantees processing reliability.

[0045] Step 120: Subscribe to and obtain the first message from the unified message queue through the business node, and inject the obtained first message into the local message queue inside the business node.

[0046] For example, subscribing to and retrieving a first message from a unified message queue through a business node, and injecting the retrieved first message into a local message queue within the business node, may include: Step 121, Subscribing to and acquiring business nodes.

[0047] As subscribers to the unified message queue, business nodes establish connections with the queue using the `JwumqConnectionObj` interface implemented in Go, and create sessions using the `JwumqSessionObj` interface. Business nodes then use the `JwumqDestinationObj` interface to create either a `PartMember` role (free mode) or a `PartPipe` role (point-to-point mode) on a specific queue, thereby subscribing to and continuously listening for messages in the unified message queue. When the first message arrives, the business node retrieves it using the `Receive` method; this process enables cross-node distribution and reception of tasks.

[0048] In this process, business nodes, acting as subscribers to the unified message queue, establish connections and communication by implementing a series of standardized Go language interfaces. For example: a connection factory is created using the `CreateConnection()` method of the `JwumqFactoryObj` interface; a connection to the unified message queue service is established using the `Connect()` method of the `JwumqConnectionObj` interface; a session is created using the `JwumqSessionObj` interface; and `JwumqDestinationObj` is used to create `PartMember` (free mode) or `PartPipe` (point-to-point mode) roles on a specific queue to subscribe to the queue. When the first message arrives, the business node retrieves the message using the `Receive()` method of the `PartMember` or `PartPipe` role.

[0049] Business nodes subscribe to a unified message queue and retrieve the first message through a standardized interface, directly establishing a decoupled communication channel between task producers and executors in the distributed system. By adopting standardized Go language interfaces such as JwumqConnectionObj and JwumqSessionObj, the system achieves standardized access between business nodes and the message queue. This allows any node conforming to the interface specification to seamlessly integrate into the system without needing to concern itself with the specific queue implementation details. This solves the problems of poor system scalability and complex node access caused by the tight coupling between communication mechanisms and business logic in the traditional BSP model, significantly improving the system's versatility and integrability.

[0050] Step 122: Injection of the first message from the unified queue to the local queue.

[0051] After receiving the first message, the business node, according to the internal architecture design requirements of the task node, injects the first message into the local message queue within the node for buffering. For example, in the Server function, the business node can obtain the first message through the e.JW[receiving node].Receive() method, create a Job object, and send it to the JobQueue (i.e., the local message queue) within the node. The local message queue, as a buffered message channel, allocates buffers according to the memory performance of each node, and can be configured independently, thus implementing temporary storage and flow control of tasks within the node.

[0052] The internal architecture includes: a unified task queue: absorbing tasks from a unified message queue; a buffered message channel: the buffer size can be configured according to the node's memory performance, used to temporarily store tasks and smooth traffic; a shared work pool: containing a buffer to control the number of concurrently executed (including waiting tasks) tasks, achieving precise concurrency management; and an alarm mechanism: monitoring the message channel and the shared work pool. An alarm is triggered when the buffer is full or congestion occurs, and data can be forwarded to other available nodes to prevent task loss and ensure system stability. In the code implementation, the Server function receives messages via e.JW[receiving node].Receive(), creates a Job object, and sends it to the node's internal JobQueue (i.e., the local message queue). The scheduler (Dispatcher) then dispatches tasks from this queue.

[0053] The first message received is buffered by injecting it into the local message queue inside the node, thus constructing a task buffering and scheduling layer within the node. Furthermore, the local message queue, acting as a buffered channel, can flexibly configure its buffer size based on the node's memory performance. This enables dynamic flow control and peak shaving between task inflow and worker processing capacity. When tasks surge, the local queue acts as a buffer to temporarily store tasks, preventing instantaneous worker overload; when tasks are sparse, workers can smoothly retrieve tasks from the queue, maintaining resource utilization. This effectively overcomes the shortcomings of the traditional PRAM model, which suffers from poor robustness and inability to cope with actual load fluctuations due to idealized synchronization assumptions, significantly improving the system's load adaptability and processing stability.

[0054] Steps 123: Coordination between the two-level queues.

[0055] The unified message queue transmits a first message (general message) containing only communication metadata. After receiving the first message, the business node needs to extract the second message (business message) and inject it into its local message queue. The scheduler (such as the Dispatcher) inside the business node then retrieves these messages from the local message queue and distributes them to workers for execution. This achieves the architectural goal of transmitting general messages in the global message queue, managing only the sender / receiver addresses, message categories, and basic attributes, while transmitting business messages in the internal message queue. This effectively solves the problem of decoupling the BSP model from business logic.

[0056] The combination of two-level queue collaboration and a two-level message encoding / decoding mechanism achieves a complete separation of communication and business logic. The unified message queue only processes the first message (i.e., the general message) carrying routing and sequence information, while the local message queue carries the second message (i.e., the business message) containing specific operational semantics. This separation allows for independent optimization of communication protocols and evolution of business logic: the unified queue can focus on improving the reliability and efficiency of cross-node communication, while business nodes can focus on optimizing task scheduling and execution algorithms. This design fundamentally solves the core problem of the difficulty in decoupling the BSP model from business logic, enabling the system architecture to simultaneously possess the high efficiency and reliability of the communication layer and the flexibility and scalability of the business layer, laying a solid technical foundation for building a general-purpose, high-performance data parallel computing system.

[0057] Step 103: Retrieve the second message from the local message queue through multiple pre-defined workers within the business node, and execute the task corresponding to the second message.

[0058] For example, workers continuously listen to the local message queue (via JobChannel). Upon receiving a task message, they execute specific business operations. After completion, they can retrieve new tasks. Multiple workers running in parallel constitute the processing capacity pool of a business node, the number of which can be fixed or dynamically adjusted according to configuration. By deploying multiple workers concurrently to retrieve tasks from the local message queue within a business node, a node-level parallel processing capacity pool is constructed. This solves the scalability bottleneck caused by the reliance on global synchronization and shared memory in the traditional PRAM model. Each worker processes tasks independently without cross-node synchronization, significantly improving the task throughput and resource utilization of a single node.

[0059] In some exemplary embodiments, the data structure of the second message is configured according to its defined task operation type, which includes at least one of the following: information caching, data modification, data query, and calculation task.

[0060] Operation types refer to the four standardized categories of the second message based on business semantics. Among them, the information caching mechanism is suitable for scenarios that require complete reading and continuous updating of data status; the data modification mechanism corresponds to data addition, deletion, and modification operations; the data query mechanism corresponds to filtering operations with complex conditions; and the computation task mechanism corresponds to complex analysis processes involving multiple data sources and computation rules. These four types cover the main business scenarios of big data processing.

[0061] By standardizing the second message into four operation types, a unified abstract interface is provided for various business processes. This enables the system to handle various scenarios, from simple data queries to complex calculation tasks, with a consistent mechanism. This overcomes the shortcomings of the BSP model, which suffers from poor versatility due to its tight coupling with specific business processes, and achieves modularity and reusability of business processes.

[0062] The second message type is precisely defined by the MsgID enumeration, covering the four business operation models mentioned above, mainly including (but not limited to): SQLREQ_V1 / SQLRESP_V1: SQL request / response messages for data query. CACHEDATAREQ_V1 / CACHEDATARESP_V1: Complete cache request / response messages for information caching. DATAUPDIND_V1 / DATAUPDACK_V1: CRUD operation messages for data modification. TASKREQ_V1 / TASKRESP_V1: Request / response messages for computation tasks, where the TASKREQV1 message body contains fields such as TaskID, TaskType, TaskFileList, and Parameter. For example, when a worker parses MsgID as TASKREQ_V1, it knows that the current task is a computation task and needs to parse the parameters according to the TASKREQV1 message structure and execute it.

[0063] In some exemplary embodiments, the method further includes: When the operation type is data query, the second message is a Structured Query Language (SQL) request message or an SQL response message; When the operation type is a computation task, the second message contains a task identifier and a list of task parameters.

[0064] SQL request messages / SQL response messages refer to the specific implementation of the second message when the operation type is data query, such as SQLREQV1 and SQLRESPV1 message structures. The request message contains a Command field carrying the SQL statement sequence, and the response message contains a Data field carrying the query result set. The two establish a request-response correspondence through the serial_number field, implementing a standardized database query interaction protocol.

[0065] In some exemplary embodiments, the system further includes a gateway node and an authentication node; wherein the node publishing the first message needs to be authenticated by the authentication node, and the message routing in the unified message queue is managed by the gateway node.

[0066] Gateway nodes are functional nodes responsible for message routing management in a unified message queue. They can be configured by a proxy server and are used for fan-in / fan-out routing to support communication in an M:N network structure. Authentication nodes are functional nodes responsible for node authentication and access control. Specifically, they can be used for authentication, general information publishing, task management, and other functional requirements, ensuring that only legitimate nodes can access the system and publish / subscribe to messages. By introducing the collaborative management of gateway and authentication nodes, communication routing optimization and access security control are achieved at the system level. The fan-in / fan-out capability of the gateway node supports a flexible M:N communication topology, while the authentication mechanism of the authentication node prevents unauthorized access. Together, they ensure the manageability, security, and stable operation of the system in a large-scale distributed environment, ultimately forming an enterprise-level data parallel computing solution that possesses both high-performance parallel processing capabilities and guaranteed security and reliability.

[0067] Specifically, in step 103, multiple workers within a business node can concurrently retrieve the second message from the local message queue according to the Dispatcher scheduling mechanism. The Dispatcher's dispatch method continuously listens to the JobQueue (local message queue), and when a task arrives, it distributes the task to an idle worker via jobChannel<- jb. Workers identify the operation type based on the MsgID defined in the second message: if it is SQLREQ_V1, they parse the SQL command in the SQLREQV1 message and execute the query; if it is TASKREQ_V1, they parse the task parameters in the TASKREQV1 message and start the computation process. Simultaneously, node access throughout the system is controlled by the authentication node, and message routing is managed by the gateway node, ensuring the security and reliability of communication.

[0068] Figure 2 A schematic diagram of the basic system architecture of the message queue-based two-level concurrency mechanism in the task processing method provided for an exemplary embodiment of this application. Figure 2 Including a unified message queue, located in Figure 2 The center, as the core hub for communication between all nodes, Figure 2 It also includes gateway nodes responsible for message routing and network topology management, authentication nodes responsible for node registration, authentication, and access control, and business nodes responsible for processing and executing specific tasks. The unified message queue is the foundation of the task processing method provided in this application embodiment. The applied distributed system adopts a master-slave server architecture, supports both point-to-point communication and broadcast subscription modes, and implements a reliable transmission mechanism of request-response-acknowledgment or transmission-acknowledgment. Figure 2The diagram shows that all nodes are interconnected through a unified message queue, forming a star topology. Task publishing nodes send task messages to the unified message queue, gateway nodes handle routing and distribution, authentication nodes verify node identities, and then business nodes retrieve tasks from the queue for processing. This architecture achieves a distributed collaboration model where nodes have equal priority.

[0069] For example, a massive amount of raw drive test data files are considered as tasks to be processed. First, the scheduling node (as the first node) encapsulates the data analysis requirements from different regions into a first message (e.g., using TASKREQ_V1 as the second message) and publishes it to a unified message queue. Business nodes (i.e., data analysis server clusters) in each provincial company subscribe to this queue. The gateway node routes the task message to the less loaded business node based on server load, and the authentication node ensures that all access nodes are legitimate analysis servers. After receiving the first message, the business node parses out the TASKREQV1 message, learns that a rasterized signal coverage analysis calculation task needs to be performed, and then injects it into its local message queue. Multiple workers (data analysis processes) pre-created within the node concurrently retrieve the task message from the queue. Each worker reads the specified drive test file, performs complex calculations such as signal strength analysis and geographic raster matching, and finally generates the analysis results. If a single drive test file is too large, the system will first decompose it into multiple atomic sub-tasks (e.g., segmented by time period), process them separately, and then aggregate them. Through the two-level message queue architecture provided in this application embodiment, the system realizes flexible scheduling and efficient utilization of cross-provincial data computing resources, solves the problems of tight coupling between business logic and resource scheduling and poor system scalability in traditional solutions, significantly improves the throughput and timeliness of big data analysis, and provides accurate data support for network optimization.

[0070] Figure 3This diagram illustrates the internal architecture of a business node (task node) in a task processing method provided as an exemplary embodiment of this application. The diagram includes a unified task queue, a buffered message channel, multiple workers, a shared work pool, and an alarm mechanism module. The unified task queue serves as the message receiving queue at the node's entry point. The buffered message channel acts as an intermediate buffer connecting the unified task queue and the workers, enabling temporary storage and flow control of tasks within the node. Multiple workers serve as processing units for concurrent task execution. The shared work pool is a control module that manages the number of concurrently executed tasks. The alarm mechanism module is an early warning system that monitors the buffer status. It provides an alarm mechanism for the message queue channel and the shared work pool when the buffer is about to fill or become congested, and forwards data to other machines to prevent task loss and ensure system stability. During processing, tasks enter the business node through a unified message queue and first enter the unified task queue, and then are assigned to a buffered message channel. The scheduler retrieves tasks from the message channel and distributes them to idle workers for execution according to the concurrency control policy of the shared work pool. When the buffer usage exceeds the threshold, an alarm mechanism is triggered, and the system can redirect tasks to other nodes to prevent task loss and system congestion.

[0071] Figure 4 This diagram illustrates the multi-level task splitting process provided by an exemplary embodiment of this application. The diagram includes the original multi-level task, a task decomposition module, three types of atomic subtasks (decomposition and scheduling tasks, split subtasks, and aggregation and verification tasks), and task execution flow arrows. The original multi-level task represents a business task with complex dependencies. The task decomposition module is the core processing unit for task decomposition, completing the core logic of converting multi-level tasks into decomposition and scheduling tasks, split subtasks, and aggregation and verification tasks. The decomposition and scheduling task is responsible for parsing the original task and scheduling and allocating subtasks. The split subtasks correspond to independent atomic tasks and are the units that actually perform specific operations. The aggregation and verification task is responsible for collecting the results of the subtasks and integrating and verifying them. During processing, the original multi-level task is first parsed by the task decomposition module to generate a decomposition and scheduling task. This task is responsible for splitting the original task into multiple independent atomic subtasks. These subtasks can be executed in parallel. After execution, the aggregation and verification task collects the results of each subtask, integrates and verifies them, and finally outputs the complete processing result. This process realizes the dismantling of the original hierarchical / causal relationship into independent atomic tasks, ensuring the reliable execution of complex tasks in a distributed environment.

[0072] The task processing method provided in this application encapsulates the task to be processed into a first message and distributes it across nodes via a unified message queue. Then, business nodes inject the first message into their local message queues. Finally, multiple workers within the business nodes concurrently retrieve and execute the corresponding second message, constructing a two-level message queue distributed parallel processing architecture. This architecture effectively solves the implementation difficulties and insufficient robustness of the traditional PRAM model due to its reliance on idealized shared memory. It achieves reliable inter-node collaboration through message communication rather than memory sharing. Simultaneously, by decoupling the business message (the second message) from the unified message format, it overcomes the shortcomings of the BSP model, such as excessive coupling with business logic and poor versatility. This enables the parallel computing system to possess both high reliability and scalability, while also flexibly adapting to diverse business processing needs, achieving a balance between high performance and versatility.

[0073] Figure 5 This is a schematic diagram of the structure of a distributed system 500 provided for an exemplary embodiment of this application. For example... Figure 5 As shown, the system 500 includes: a unified message queue 501 and at least one business node 502, wherein: Unified message queue 501 is used to transmit the first message, which is encapsulated by the task to be processed. At least one service node 502 includes: Message interface unit 5021 is used to obtain a first message from the unified message queue and inject it into the local message queue; Local message queue 5022 is used to buffer the first message from the message interface unit; Multiple workers 5023 are used to retrieve a second message from the local message queue and execute the task corresponding to the second message.

[0074] Distributed systems 500 can achieve Figures 1-4 For details of the method implementation examples, please refer to [link / reference]. Figures 1-4 The task processing method of the illustrated embodiment will not be described in detail again.

[0075] Figure 6 This is a schematic diagram of the structure of a task processing apparatus 600 provided for an exemplary embodiment of this application. (See attached diagram.) Figure 6 As shown, the device 600 includes: a message publishing module 610, a message injection module 620, and a task execution module 630, wherein: The message publishing module 610 is used to encapsulate the task to be processed into a first message and publish it to the unified message queue of the system. The message injection module 620 is used to subscribe to and obtain the first message from the unified message queue through the business node, and to inject the obtained first message into the local message queue inside the business node. The task execution module 630 is used to obtain a second message from the local message queue through multiple workers pre-set within the business node, and execute the task corresponding to the second message.

[0076] The task processing apparatus provided in this application constructs a two-level message queue distributed parallel processing architecture by encapsulating the task to be processed into a first message and distributing it across nodes via a unified message queue, then having the business node inject it into its local message queue, and finally having multiple workers within the business node concurrently obtain and execute the corresponding second message. This architecture effectively solves the problems of implementation difficulties and insufficient robustness caused by the reliance on idealized shared memory in the traditional PRAM model. It achieves reliable inter-node collaboration through message communication rather than memory sharing. At the same time, it decouples the business message (i.e., the second message) from the unified message format, overcoming the defects of high coupling and poor versatility of the BSP model with business logic. This makes the parallel computing system not only highly reliable and scalable, but also flexibly adaptable to diverse business processing needs, achieving a balance between high performance and versatility.

[0077] Optionally, the first message encapsulates the encoded data of the second message.

[0078] Optionally, the data structure of the first message includes: a message type field, a message sequence number field, a message classification identifier field, and a message payload field; the message payload field is used to carry the encoded data of the second message.

[0079] Optionally, the data structure of the second message is configured according to its defined task operation type, which includes at least one of the following: information caching, data modification, data query, and calculation task.

[0080] Optionally, when the operation type is a data query, the second message is a Structured Query Language (SQL) request message or an SQL response message; When the operation type is a computation task, the second message includes a task identifier and a list of task parameters.

[0081] Optionally, the task encapsulated by the first message and transmitted through the unified message queue is atomic; the apparatus further includes: The decomposition module is used to decompose the task to be processed into multiple independent atomic subtasks before the task to be processed is encapsulated into a first message. An encapsulation module is used to encapsulate each atomic subtask into a plurality of corresponding first messages for each atomic subtask, in order to execute subsequent steps.

[0082] The task processing unit 600 is capable of achieving Figures 1-4 For details of the method implementation examples, please refer to [link / reference]. Figures 1-4 The task processing method of the illustrated embodiment will not be described in detail again.

[0083] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. For example... Figure 7 As shown, the device includes a memory 71 and a processor 72.

[0084] Memory 71 is used to store computer programs and can be configured to store various other data to support operation on the computing device. Examples of this data include instructions for any application or method used to operate on the computing device, contact data, phone book data, messages, images, videos, etc.

[0085] The processor 72, coupled to the memory 71, is used to execute a computer program in the memory 71 for: encapsulating a task to be processed into a first message and publishing it to the unified message queue of the system; subscribing to and obtaining the first message from the unified message queue through a business node, and injecting the obtained first message into a local message queue inside the business node; obtaining a second message from the local message queue through multiple workers pre-set within the business node, and executing the task corresponding to the second message.

[0086] The electronic device provided in this application constructs a two-level message queue distributed parallel processing architecture by encapsulating the task to be processed into a first message and distributing it across nodes via a unified message queue, then having the business nodes inject it into their local message queues, and finally having multiple workers within the business nodes concurrently obtain and execute the corresponding second message. This architecture effectively solves the problems of implementation difficulties and insufficient robustness caused by the reliance on idealized shared memory in the traditional PRAM model. It achieves reliable inter-node collaboration through message communication rather than memory sharing. At the same time, it decouples the business message (i.e., the second message) from the unified message format, overcoming the defects of the BSP model, which has excessive coupling with business logic and poor versatility. This makes the parallel computing system not only highly reliable and scalable, but also flexibly adaptable to diverse business processing needs, achieving a balance between high performance and versatility.

[0087] Furthermore, such as Figure 7 As shown, the electronic device also includes other components such as a communication component 73, a display 74, a power supply component 75, and an audio component 76. Figure 7The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 7 The components shown. Additionally, depending on the implementation of the traffic playback device, Figure 7 The components within the dashed box are optional, not mandatory. For example, when an electronic device is implemented as a terminal device such as a smartphone, tablet, or desktop computer, it may include... Figure 7 The components within the dashed box; when the electronic device is implemented as a server-side device such as a conventional server, cloud server, data center, or server array, it may be excluded. Figure 7 The component within the dashed box.

[0088] The above Figure 7 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may further include a Near Field Communication (NFC) module, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, etc.

[0089] The above Figure 7 The memory in the memory can be implemented by any class of volatile or non-volatile storage devices or combinations thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0090] The above Figure 7 The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action, but also the duration and pressure associated with the touch or swipe operation.

[0091] The above Figure 7 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0092] The above Figure 7 The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0093] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described task processing method embodiments.

[0094] Accordingly, this application also provides a computer program product, which stores instructions that, when executed by a computer, cause the computer to perform the steps in the task processing method embodiment provided in this application.

[0095] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0098] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0100] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A task processing method, characterized in that, The method is applied to a distributed system containing multiple nodes, including: The task to be processed is encapsulated into a first message and published to the unified message queue of the system. The first message is subscribed to and retrieved from the unified message queue through the business node, and the retrieved first message is injected into the local message queue inside the business node. The second message is retrieved from the local message queue by multiple pre-set workers within the business node, and the task corresponding to the second message is executed.

2. The method as described in claim 1, characterized in that, The first message encapsulates the encoded data of the second message.

3. The method as described in claim 1 or 2, characterized in that, The data structure of the first message includes: a message type field, a message sequence number field, a message classification identifier field, and a message payload field; the message payload field is used to carry the encoded data of the second message.

4. The method as described in claim 1, characterized in that, The data structure of the second message is configured according to its defined task operation type, which includes at least one of the following: information caching, data modification, data query, and calculation task.

5. The method as described in claim 4, characterized in that, When the operation type is data query, the second message is a Structured Query Language (SQL) request message or an SQL response message; When the operation type is a computation task, the second message includes a task identifier and a list of task parameters.

6. The method as described in claim 1, characterized in that, The task encapsulated by the first message and transmitted through the unified message queue is atomic; the method further includes: Before encapsulating the task to be processed into a first message, the task to be processed is decomposed into multiple independent atomic subtasks. For each of the atomic subtasks, the atomic task is encapsulated into a plurality of corresponding first messages for execution of subsequent steps.

7. A distributed system, characterized in that, The system includes: A unified message queue is used to transmit the first message, which is encapsulated by the task to be processed. At least one business node, including: The message interface unit is used to obtain the first message from the unified message queue and inject it into the local message queue; A local message queue is used to buffer the first message from the message interface unit; Multiple workers are used to retrieve a second message from the local message queue and execute the task corresponding to the second message.

8. A task processing device, characterized in that, include: The message publishing module is used to encapsulate the tasks to be processed into a first message and publish it to the unified message queue of the system. The message injection module is used to subscribe to and obtain the first message from the unified message queue through the business node, and to inject the obtained first message into the local message queue inside the business node. The task execution module is used to obtain a second message from the local message queue through multiple workers pre-set within the business node, and execute the task corresponding to the second message.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.

11. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the steps of the method as described in any one of claims 1 to 6.

Images