Task queue-based distributed computing method and system and intelligent terminal

Abstract

The application provides a distributed computing method and system based on a task queue and an intelligent terminal, and comprises the following steps: forming an index list according to read index information; obtaining an index calculation task and forming a task queue according to indexes in the index list; receiving the index calculation task through the task queue by using a distributed computing engine and calculating data results, and pushing the data results to a data queue; and pushing the data results in the data queue into a data center for storage of index results. The application realizes a horizontally expandable distributed index calculation platform through bidirectional dimensions of contract code and index algorithm, task splitting and distributed scheduling through a task queue.

Description

Technical Field

[0001] This invention relates to the field of distributed task scheduling, and more specifically, to a distributed computing method and system based on task queues, as well as a smart terminal. Background Technology

[0002] With the rise of quantitative technology in the financial industry, more and more quantitative indicators are being applied to the stock market, leading to a growing demand for quantitative computing. Traditional quantitative indicators are calculated by a single program. However, due to the large number of contracts in the market and the real-time changes in stock prices, the computational pressure on a single process becomes too great. Therefore, the trend is towards multi-process distributed computing, but the challenge lies in solving the problem of distributed scheduling.

[0003] The conventional approach is to allocate tasks across multiple processes within a single metric program, thus fully utilizing the computing power of multi-core CPUs. However, this couples the quantification algorithm with the engineering process, making the development of quantification metrics more complex.

[0004] Therefore, in order to reduce the development cost of quantization algorithms and improve the quantization computing capabilities, it is necessary to split tasks through a two-way dimension of contract code and indicator algorithms, and to use task queues for distributed scheduling, thus realizing a horizontally scalable distributed indicator computing platform. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a distributed computing method and system based on task queues, as well as a smart terminal.

[0006] A distributed computing method based on a task queue, provided by the present invention, includes:

[0007] Indicator management step S1: Generate an indicator list based on the read indicator information;

[0008] Task scheduling step S2: Based on the indicators in the indicator list, calculate the indicator tasks and form a task queue;

[0009] Distributed computing engine processing step S3: Using a distributed computing engine, the system receives metric calculation tasks through a task queue, calculates the data results, and pushes the data results to the data queue.

[0010] Indicator storage step S4: Push the data results in the data queue into the data center to store the indicator results.

[0011] Preferably, the task scheduling step S2 includes:

[0012] Step S2.1: For each indicator in the indicator list, read the indicator calculation strategy for each period.

[0013] Step S2.2: For each indicator in the indicator list, determine the applicable market scope, and read the full list of market contracts based on the market scope;

[0014] Step S2.3: Combine the full list of market contracts with the corresponding indicator calculation strategies to generate the smallest granularity indicator calculation task;

[0015] Step S2.4: Using the contract code in the full list of market contracts as the key, put the indicator calculation task into the task queue; wherein, the elements in the task queue are partitioned by contract code.

[0016] Preferably, the distributed computing engine processing step S3 includes:

[0017] Step S3.1: The distributed computing engine contains a variable number of nodes. The number of nodes in the distributed computing engine is dynamically adjusted based on the task queue backlog. Each node uses contract code to find a specific partition and obtains the metric calculation task from that partition.

[0018] Step S3.2: Extract the corresponding indicator information of the indicator algorithm task, load the remote indicator algorithm source code through the indicator information, and generate a localized module based on the indicator algorithm source code;

[0019] Step S3.3: Pass the elements in the task queue as parameters into the algorithm program corresponding to the index algorithm source code to calculate the index and obtain the data results;

[0020] Step S3.4: Verify the validity of the data results, and then place the data results into the data queue according to the position indicated by the contract code.

[0021] Preferably, in the task scheduling step S2, tasks are split using contract code and metric calculation strategies, and distributed scheduling is performed using task queues.

[0022] A distributed computing system based on a task queue, according to the present invention, includes:

[0023] Indicator Management Module M1: Generates an indicator list based on the read indicator information;

[0024] Task scheduling module M2: Based on the indicators in the indicator list, it obtains indicator calculation tasks and forms a task queue;

[0025] Distributed computing engine processing module M3: It adopts a distributed computing engine, receives indicator calculation tasks through a task queue, calculates the data results, and pushes the data results to the data queue.

[0026] Indicator storage module M4: Pushes the data results in the data queue into the data center for storage of indicator results.

[0027] Preferably, the task scheduling module M2 includes:

[0028] Module M2.1: For each indicator in the indicator list, read the indicator calculation strategy for each period.

[0029] Module M2.2: For each indicator in the indicator list, determine the applicable market scope and read the full list of market contracts based on the market scope;

[0030] Module M2.3: Combines the full list of market contracts with the corresponding indicator calculation strategies to generate the smallest granularity indicator calculation task;

[0031] Module M2.4: Using the contract code in the full list of market contracts as the key, the indicator calculation task is placed into the task queue; wherein, the elements in the task queue are partitioned by contract code.

[0032] Preferably, the distributed computing engine processing module M3 includes:

[0033] Module M3.1: The distributed computing engine contains a variable number of nodes, and the number of nodes in the distributed computing engine is dynamically adjusted according to the task queue backlog; a single node finds a specific partition through contract code and obtains the metric calculation task from that partition;

[0034] Module M3.2: Extracts the corresponding indicator information of the indicator algorithm task, loads the remote indicator algorithm source code through the indicator information, and generates a localized module based on the indicator algorithm source code;

[0035] Module M3.3: Passes the elements in the task queue as parameters into the algorithm program corresponding to the indicator algorithm source code, performs indicator calculation, and obtains data results;

[0036] Module M3.4: Performs validity verification on the data results, and then places the data results into the data queue according to the position indicated by the contract code.

[0037] Preferably, in the task scheduling module M2, tasks are split according to contract code and indicator calculation strategy, and distributed scheduling is performed through task queues.

[0038] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the steps of the task queue-based distributed computing method are implemented.

[0039] According to the present invention, a smart terminal includes the aforementioned task queue-based distributed computing system, or includes the aforementioned computer-readable storage medium storing computer programs.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This invention splits tasks through a two-way dimension of contract code and indicator algorithm, and performs distributed scheduling through task queues, thereby realizing a horizontally scalable distributed indicator calculation platform.

[0042] 2. In the development of indicators, this invention only requires focusing on the algorithm implementation. After the algorithm is completed, it is placed into the task management for calculation and scheduling. There is no need to pay attention to the calculation of massive amounts of data, which saves time and energy for repeated engineering development.

[0043] 3. The key technical point of this invention is to split tasks from the dimension of contract code and perform distributed scheduling through task queues. The distributed computing engine consumes different task queues and dynamically loads indicator algorithms for calculation, thereby realizing the distributed calculation of quantitative indicators. Attached Figure Description

[0044] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0045] Figure 1 This is a schematic diagram illustrating the working principle of the present invention.

[0046] Figure 2 This is a schematic diagram of the overall process steps of the present invention.

[0047] Figure 3 This is a schematic diagram of the process steps of step S2 of the present invention.

[0048] Figure 4 This is a schematic diagram of the process steps of step S3 of the present invention. Detailed Implementation

[0049] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0050] To address the aforementioned issues, this invention implements a distributed computing method and system based on task queues. In the development of metrics, focus is placed solely on algorithm implementation; once the algorithm is complete, it is placed into task management for computation scheduling. There is no need to concern oneself with the computation of massive amounts of data, saving time and effort spent on repeated engineering development.

[0051] Figure 1 The overall framework of the present invention is shown, which includes indicator management, task scheduling, distributed computing engine, and indicator data center.

[0052] like Figure 2 As shown, the distributed computing method based on task queues provided by this invention specifically includes:

[0053] Indicator Management Step S1: Read indicator information and obtain the indicator's version number and unique code; for example, the indicator is a financial indicator such as a stock. Specifically, read indicator information from the indicator database, which includes information on each version of the indicator, each periodic task of the indicator, and the task scheduling strategy.

[0054] Task scheduling step S2: Generate indicator calculation tasks. This step combines tasks based on the full market contract code and sends the generated tasks to the task queue, awaiting execution by the distributed computing engine. A single indicator may require calculations over multiple periods and is applied to all market contracts. Here, indicators and contracts are combined into the smallest unit to form fine-grained task entities. Specifically, such as... Figure 3 As shown, task scheduling step S2 specifically includes the following: Step S2.1: Based on the indicator list, iteratively read the indicator calculation strategy for each indicator in each period; Step S2.2: Based on the indicator read in the previous step, determine its applicable market scope and read the full list of market contracts for subsequent use; Step S2.3: Apply the contract list obtained in the previous step and combine it with the specific indicator calculation strategy to generate the smallest granularity computing task; Step S2.4: Using the contract code as the key, put the task generated in the previous step into the task queue. In this step, the task queue will be partitioned by contract code to improve distributed computing capabilities.

[0055] Distributed computing engine processing step S3: The distributed computing engine receives metric tasks through a task queue, calculates the metrics, obtains the metric results, and pushes the results to the data queue. The distributed computing engine itself does not contain algorithm capabilities; it dynamically loads the metric algorithm source code through metric tasks for calculation, and then pushes the calculated data to the data center. Here, the loaded source code is cached locally to improve performance. Specifically, as follows... Figure 4As shown, step S33 of the distributed computing engine mainly includes the following: Step S3.1: The distributed computing engine contains a variable number of nodes, which can be dynamically adjusted based on task queue pressure. A single node uses contract code to find a specific partition and obtains an indicator calculation task from that partition; Step S3.2: From the task obtained in the previous step, information about the indicator algorithm itself, such as the version number and unique code, is extracted. This information is used to load the indicator algorithm source code from the remote repository and generate a localized module, avoiding repeated resource retrieval from the remote end; Step S3.3: The task elements are passed as parameters into the algorithm program for indicator calculation; Step S3.4: The data results obtained in the previous step are validated, and then the indicator data is placed into the data queue according to the contract code.

[0056] Metric storage step S4: Push the metric calculation results into the data center for storage. The data center will categorize data by metric, and store the same metric by time dimension.

[0057] The present invention also provides a distributed computing system based on task queues. Those skilled in the art can implement the distributed computing system based on task queues by executing the process steps of the distributed computing method based on task queues. That is, the distributed computing method based on task queues can be understood as a preferred embodiment of the distributed computing system based on task queues.

[0058] A distributed computing system based on a task queue, according to the present invention, includes:

[0059] Indicator Management Module M1: Generates an indicator list based on the read indicator information;

[0060] Task scheduling module M2: Based on the indicators in the indicator list, it obtains indicator calculation tasks and forms a task queue;

[0061] Distributed computing engine processing module M3: It adopts a distributed computing engine, receives indicator calculation tasks through a task queue, calculates the data results, and pushes the data results to the data queue.

[0062] Indicator storage module M4: Pushes the data results in the data queue into the data center for storage of indicator results.

[0063] The task scheduling module M2 includes:

[0064] Module M2.1: For each indicator in the indicator list, read the indicator calculation strategy for each period.

[0065] Module M2.2: For each indicator in the indicator list, determine the applicable market scope and read the full list of market contracts based on the market scope;

[0066] Module M2.3: Combines the full list of market contracts with the corresponding indicator calculation strategies to generate the smallest granularity indicator calculation task;

[0067] Module M2.4: Using the contract code in the full list of market contracts as the key, the indicator calculation task is placed into the task queue; wherein, the elements in the task queue are partitioned by contract code.

[0068] The distributed computing engine processing module M3 includes:

[0069] Module M3.1: The distributed computing engine contains a variable number of nodes, and the number of nodes in the distributed computing engine is dynamically adjusted according to the task queue backlog; a single node finds a specific partition through contract code and obtains the metric calculation task from that partition;

[0070] Module M3.2: Extracts the corresponding indicator information of the indicator algorithm task, loads the remote indicator algorithm source code through the indicator information, and generates a localized module based on the indicator algorithm source code;

[0071] Module M3.3: Passes the elements in the task queue as parameters into the algorithm program corresponding to the indicator algorithm source code, performs indicator calculation, and obtains data results;

[0072] Module M3.4: Performs validity verification on the data results, and then places the data results into the data queue according to the position indicated by the contract code.

[0073] In the task scheduling module M2, tasks are split according to contract code and indicator calculation strategy, and distributed scheduling is performed through task queues.

[0074] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0075] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the steps of the task queue-based distributed computing method are implemented.

[0076] According to the present invention, a smart terminal includes the aforementioned task queue-based distributed computing system, or includes the aforementioned computer-readable storage medium storing computer programs.

[0077] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A distributed computing method based on task queues, characterized in that, include: Indicator management step S1: Generate an indicator list based on the read indicator information; Task scheduling step S2: Based on the indicators in the indicator list, calculate the indicator tasks and form a task queue; Distributed computing engine processing step S3: Using a distributed computing engine, the system receives metric calculation tasks through a task queue, calculates the data results, and pushes the data results to the data queue. Indicator storage step S4: Push the data results in the data queue into the data center to store the indicator results; The task scheduling step S2 includes: Step S2.1: For each indicator in the indicator list, read the indicator calculation strategy for each period. Step S2.2: For each indicator in the indicator list, determine the applicable market scope, and read the full list of market contracts based on the market scope; Step S2.3: Combine the full list of market contracts with the corresponding indicator calculation strategies to generate the smallest granularity indicator calculation task; Step S2.4: Using the contract code in the full list of market contracts as the key, put the indicator calculation task into the task queue; wherein, the elements in the task queue are partitioned by contract code.

2. The distributed computing method based on task queues according to claim 1, characterized in that, The distributed computing engine processing step S3 includes: Step S3.1: The distributed computing engine contains a variable number of nodes. The number of nodes in the distributed computing engine is dynamically adjusted based on the task queue backlog. Each node uses contract code to find a specific partition and obtains the metric calculation task from that partition. Step S3.2: Extract the corresponding indicator information of the indicator calculation task, load the remote indicator algorithm source code through the indicator information, and generate a localized module based on the indicator algorithm source code; Step S3.3: Pass the elements in the task queue as parameters into the algorithm program corresponding to the index algorithm source code to calculate the index and obtain the data results; Step S3.4: Verify the validity of the data results, and then place the data results into the data queue according to the position indicated by the contract code.

3. The distributed computing method based on task queues according to claim 1, characterized in that, In the task scheduling step S2, tasks are split using contract code and metric calculation strategies, and distributed scheduling is performed using task queues.

4. A distributed computing system based on task queues, characterized in that, include: Indicator Management Module M1: Generates an indicator list based on the read indicator information; Task scheduling module M2: Based on the indicators in the indicator list, it obtains indicator calculation tasks and forms a task queue; Distributed computing engine processing module M3: It adopts a distributed computing engine, receives indicator calculation tasks through a task queue, calculates the data results, and pushes the data results to the data queue. Indicator storage module M4: Pushes the data results in the data queue into the data center for storage of indicator results; The task scheduling module M2 includes: Module M2.1: For each indicator in the indicator list, read the indicator calculation strategy for each period. Module M2.2: For each indicator in the indicator list, determine the applicable market scope and read the full list of market contracts based on the market scope; Module M2.3: Combines the full list of market contracts with the corresponding indicator calculation strategies to generate the smallest granularity indicator calculation task; Module M2.4: Using the contract code in the full list of market contracts as the key, the indicator calculation task is placed into the task queue; wherein, the elements in the task queue are partitioned by contract code.

5. The distributed computing system based on task queues according to claim 4, characterized in that, The distributed computing engine processing module M3 includes: Module M3.1: The distributed computing engine contains a variable number of nodes, and the number of nodes in the distributed computing engine is dynamically adjusted according to the task queue backlog; a single node finds a specific partition through contract code and obtains the metric calculation task from that partition; Module M3.2: Extracts the corresponding indicator information of the indicator calculation task, loads the remote indicator algorithm source code through the indicator information, and generates a localized module based on the indicator algorithm source code; Module M3.3: Passes the elements in the task queue as parameters into the algorithm program corresponding to the indicator algorithm source code, performs indicator calculation, and obtains data results; Module M3.4: Performs validity verification on the data results, and then places the data results into the data queue according to the position indicated by the contract code.

6. The distributed computing system based on task queues according to claim 4, characterized in that, In the task scheduling module M2, tasks are split according to contract code and indicator calculation strategy, and distributed scheduling is performed through task queues.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the distributed computing method based on task queues as described in any one of claims 1 to 3.

8. A smart terminal, characterized in that, It includes the distributed computing system based on task queues as described in any one of claims 4 to 6, or the computer-readable storage medium storing a computer program as described in claim 7.

Images