Stock selection system based on distributed efficient index formula

Abstract

The application provides a stock selection system based on a distributed efficient index formula, comprising a front-end agent service module, a plurality of sharding calculation service modules and a data source service module. The front-end agent service module receives a user request containing a plate factor and an index formula, acquires a benchmark stock pool list requiring stock selection according to the plate factor, divides the benchmark stock pool list requiring stock selection into a plurality of shards by taking the modulus of an incremental internal code and the total number of shards, and generates a corresponding shard calculation request; a shard calculation node management module in the front-end agent service module distributes the request to the corresponding shard calculation service module according to the shard identifier by using an adaptive consistent hashing algorithm. The application separates the front-end agent and the shard calculation by combining the MapReduce task distribution and the result aggregation mechanism, improves the real-time performance and the concurrent processing capability of stock selection calculation, dynamically adjusts the distribution of virtual nodes and entity nodes by using the adaptive consistent hashing strategy, solves the problem of unbalanced sharding, and supports node elastic scaling.

Description

Technical Field

[0001] This invention relates to the field of financial data processing technology, and more specifically, to a system for stock selection based on distributed, efficient indicator formulas. Background Technology

[0002] In the field of financial investment, as the stock market continues to expand and the complexity of trading data continues to increase, quickly and accurately screening out targets that meet specific indicators from a massive amount of stocks has become a core need for investors and financial institutions.

[0003] Current traditional stock selection systems mostly adopt a monolithic service architecture, which has many limitations in practical applications: First, its computational efficiency is low. When faced with the computational needs of large-scale stock pools or complex indicator formulas, a single computing node cannot handle the excessive computational pressure, directly resulting in excessively long stock selection time and failing to meet the real-time requirements of the scenario. At the same time, the scalability of the monolithic service architecture is significantly limited, and it cannot achieve parallel expansion. In high-concurrency scenarios, service congestion is prone to occur, which in turn affects the user experience. In addition, although some traditional stock selection systems have attempted to adopt distributed architecture optimization, they still rely on the traditional consistent hash algorithm for stock sharding. However, the number of stocks in the entire market is limited, and this algorithm cannot achieve balanced sharding when the amount of data is insufficient. This ultimately leads to an imbalance where some computing nodes are overloaded and others are idle. Moreover, the indicator formula calculation logic of traditional stock selection systems is highly coupled with the service architecture, resulting in insufficient customization capabilities. When users need to customize indicators according to their own needs, the entire system needs to be completely transformed, which not only prolongs the function launch cycle but also lacks sufficient flexibility, making it difficult to adapt to diverse stock selection needs.

[0004] A patent literature search revealed an invention patent with publication number CN118519780A, which discloses a cloud-based distributed quantitative strategy development platform. This platform comprises a factor development and analysis module, a machine learning model development and analysis module, a machine learning model integration module, an asset portfolio strategy development and analysis optimization module, a distributed cloud computing engine module, and a management module. It boasts high reliability and scalability, utilizing distributed cloud computing technology combined with microservice architecture and containerization to support high-concurrency data processing and rapid execution of complex algorithms. It covers factor development and analysis functions, machine learning model development, analysis and integration functions, asset portfolio strategy development and analysis optimization functions, distributed computing engine functions, and performance evaluation functions, effectively improving the development efficiency and execution performance of asset investment strategies. However, this patent fails to address the technical problems of low computational efficiency, uneven sharding, and weak customization. It also fails to decouple indicator calculation from the architecture, making it difficult to meet the needs of efficient stock selection.

[0005] In summary, given the problems of the existing technologies, researching a stock selection system based on distributed and efficient indicator formulas has become a critical task that urgently needs to be addressed. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a stock selection system based on a distributed, efficient indicator formula.

[0007] The present invention provides a system for stock selection based on distributed high-efficiency indicator formulas, comprising: a front-end agent service module, multiple sharded computing service modules, and a data source service module;

[0008] The front-end proxy service module receives user requests, which include the sector factors and indicator formulas to be calculated. Based on the sector factors, the front-end proxy service module obtains the corresponding stock pool list as the benchmark stock pool list for stock selection. It then iterates through the benchmark stock pool list, performing a modulo operation based on the incremental internal code of each stock and the total number of shards to determine the shard to which each stock belongs, generating multiple shard calculation requests. Each shard calculation request contains a sub-stock pool list corresponding to a shard and a shard identifier. Finally, through its internal shard calculation node management module, the front-end proxy service module distributes the multiple shard calculation requests to the corresponding shard calculation service modules using an adaptive consistent hashing method based on the shard identifier.

[0009] The sharding calculation service module is used to receive sharding calculation requests; the sharding calculation service module obtains the corresponding stock information parameters from the data source service module according to the sharding identifier in the sharding calculation request; the sharding calculation service module calls the embedded indicator stock selection SDK module, inputs the stock information parameters into the preset indicator formula calculation function for processing, obtains the stock screening results and returns them to the front-end agent service module.

[0010] The data source service module is used to provide stock information parameters to the sharding calculation service module;

[0011] The front-end agent service module aggregates the stock screening results returned by multiple sharded computing service modules to generate the final stock selection results.

[0012] Preferably, the process by which the front-end proxy service module generates multiple sharding computation requests includes:

[0013] Step a1: The front-end agent service module internally maintains an incremental internal code mapping table for the entire A-share pool. The incremental internal code mapping table stores the correspondence between stock codes and incremental internal codes.

[0014] Step a2: The front-end agent service module subscribes to the change notification of the entire market A-share pool list and dynamically maintains the incremental internal code in the incremental internal code mapping table according to the stock code sorting.

[0015] Step a3: The front-end proxy service module traverses the list of benchmark stock pools that need to be selected. For each stock in the list of benchmark stock pools that need to be selected, it performs a modulo operation based on the incremental internal code of the stock and the total number of shards to determine the shard identifier to which the stock belongs.

[0016] Step a4: The front-end agent service module traverses all shard identifiers, summarizes the list of stocks belonging to the same shard identifier, and forms a sub-stock pool list corresponding to each shard identifier.

[0017] Preferably, the process by which the front-end proxy service module distributes multiple shard computing requests based on the shard identifier using an adaptive consistent hashing method through its internal shard computing node management module includes:

[0018] Step b1: The front-end proxy service module iterates through all shard identifiers and constructs a shard calculation request for each shard identifier. The shard calculation request includes the core fields of the user request, the shard identifier, and the corresponding sub-stock pool list.

[0019] In step b2, the front-end proxy service module concurrently submits multiple sharding computation requests to the sharding computation node management module through an internally maintained thread pool.

[0020] Step b3: The sharded compute node management module performs routing forwarding for each sharded compute request through its internally maintained scheduler. Routing forwarding includes:

[0021] Step b3.1: The scheduler maintains a hash ring, on which all sharded computing service nodes and their associated virtual nodes are mapped.

[0022] Step b3.2: For each received sharding computation request, the scheduler calculates a hash value based on the sharding identifier in the sharding computation request, and determines the target sharding computation service node on the hash ring based on the hash value;

[0023] In step b3.3, the scheduler forwards the sharding computation request to the target sharding computation service node.

[0024] Preferably, the sharded computing node management module dynamically adjusts the number of physical nodes or the distribution of virtual nodes participating in the hash ring based on the real-time load status of each sharded computing service node using an adaptive consistent hashing method. The adaptive consistent hashing method includes an adaptive consistent hashing expansion node algorithm and an adaptive consistent hashing shrink node algorithm.

[0025] Preferably, the adaptive consistent hashing resizing node algorithm is as follows:

[0026] DCHEN(S(t),Θ):

[0027]

[0028] in,

[0029] DCHEN is the adaptive consistent hashing node expansion processing function;

[0030] S(t) = A set of real-time node states;

[0031] This represents the current CPU utilization of the i-th shard computing service node;

[0032] This represents the current Mem memory usage of the i-th shard computing service node;

[0033] Θ={ , , ,start,end, , Configure parameters for the service system;

[0034] This is the upper limit threshold for Mem memory;

[0035] This is the CPU upper limit threshold;

[0036] The default number of virtual nodes is added when expanding the number of physical nodes;

[0037] The adjustment step size is the volatility that changes when the virtual node is scaled down.

[0038] `start` is the start time for the DCHEN function to execute; the default is the opening time.

[0039] `end` is the termination time for the DCHEN function, with the default closing time.

[0040] The maximum number of service nodes for a sharded computing node cluster;

[0041] This is a function to expand entity nodes for global overload.

[0042] This is a function for shrinking virtual nodes to handle single-point overload.

[0043] To record the relevant status data of each node in the disk;

[0044] The adaptive node expansion factor is defined as:

[0045]

[0046] The calculation method is to iterate through all nodes and calculate the percentage of CPU or Mem that exceeds the threshold.

[0047] in,

[0048] The adaptive consistent hashing algorithm for expanding nodes includes the following steps:

[0049] In step C1, the sharded computing node management module within the front-end agent service module asynchronously executes the DCHEN(S(t),Θ) ​​function based on the system parameters start and end times, and performs real-time calculations based on S(t). The value is used to perform timed judgment and processing of the expansion process;

[0050] Step C2, when ≥0.5 ,trigger Expand node operation;

[0051] Step C3, when When triggered Perform a shrinkage operation on the specified virtual node;

[0052] Step C4: If the conditions of steps C2 and C3 are not met, execute... The function performs a node state storage procedure, recording the state of each shard computing node to disk according to time points.

[0053] Preferably, in step C2 The node expansion operation includes the following sub-steps:

[0054] Step C2.1: The sharded compute node management module inside the front-end agent service module notifies the node operation through the service bus, performs the expansion operation through the K8S API, and completes the registration of the new sharded compute node service.

[0055] Step C2.2, New entity node After automatic expansion, the entity nodes will be... of Number of virtual nodes and entity nodes It is added to the hash ring itself;

[0056] Step C2.3: Upon the arrival of a new sharding computation request, entity nodes are automatically reallocated. and virtual nodes Surrounding sharded data has not been assigned to entity nodes. and virtual nodes The fragmented data continues to fall to the previous node.

[0057] Preferably, in step C3 Performing a virtual node scaling-down operation on a specified node includes the following sub-steps:

[0058] Step C3.1: The sharding node management module within the front-end agent service module iterates through the list of all nodes exceeding the threshold. ];

[0059] Step C3.2, list the nodes that exceed the threshold [ Adaptively reduce the number of virtual nodes according to the shrinkage function. Calculate the number of new virtual nodes for each node in the list of nodes that exceed the threshold;

[0060] Step C3.3: Upon the arrival of a new sharding computation request, the list of nodes exceeding the threshold is automatically reallocated. The fragmented data.

[0061] Preferably, the adaptive consistent hashing shrinking node algorithm is as follows:

[0062] DCHSN( ,Θ):

[0063]

[0064] in,

[0065] DCHSN is the adaptive consistent hash shrinking node processing function;

[0066] This is a collection of the states of each node during the latest transaction period;

[0067] ={S(t)|t∈{ , ..., }};

[0068] S(t) = / / A set of node states at a specified time point;

[0069] Θ={ , , , ,α,β,start,end,collectTime} are the service system configuration parameters; Minimum number of service nodes for a sharded computing node cluster; The fluctuation rate when expanding the capacity of virtual nodes is the adjustment step size. This is the lower limit threshold for CPU usage. α is the lower limit threshold for Mem memory usage; α is the CPU computation ratio during scaling down computation; β is the Mem computation ratio during scaling down computation; start is... The start time of the statistical node status set is the default opening time; end is the end time of the statistical node status set is the default closing time; collectTime is the time point for the DCHSN function to process the data.

[0070] The adaptive node shrinkage factor is defined as follows:

[0071]

[0072] The calculation method involves iterating through all nodes and calculating the percentage of CPU usage for each node whose Mem is below a threshold.

[0073] This is a function for globally low-load shrinking entity nodes;

[0074] Functions for expanding virtual nodes with low load at a single point;

[0075] The adaptive consistent hashing shrinking node algorithm includes the following steps:

[0076] Step D1: The sharded computing node management module within the front-end agent service module asynchronously executes DCHSN based on the system parameter collectTime. The function ,Θ) is based on Calculate the day The value is used to perform timed judgment and processing of the scaling down process;

[0077] Step D2, when When triggered Reduce node operations

[0078] Step D3, when When triggered Perform virtual node expansion operations on the specified node.

[0079] Preferably, in step D2 The node reduction operation includes the following sub-steps:

[0080] In step D2.1, when the sharded computing node management module inside the front-end agent service module executes the node reduction function, it iterates through... The average state value of each node is calculated and the node with the smallest percentage is identified. ,calculate:

[0081] = ,

[0082] in,

[0083] This represents the CPU utilization of the j-th shard computing service node at time t;

[0084] This represents the memory usage of Mem at time t for the j-th shard computing service node;

[0085] for The total number of time points collected in the set is used to calculate the target node with the lowest average CPU and Mem utilization based on this formula.

[0086] Step D2.2: The internal node management module of the front-end agent service module notifies the node cleanup operation through the service bus and performs node shrinking operation through the K8S API.

[0087] Step D2.3, After the node is successfully cleaned up, the node is triggered. Related virtual nodes and nodes It is removed from the hash ring;

[0088] Step D2.4: Upon receiving a new sharding computation request, nodes are automatically reallocated. and virtual nodes The surrounding sharded data is transferred to other nodes for processing.

[0089] Preferably, in step D3 Performing a virtual node expansion operation for a specified node includes the following sub-steps:

[0090] Step D3.1: The sharding computing node management module within the front-end proxy service module finds all the sharded computing nodes by traversing the process. List of nodes below the threshold [ ];

[0091] Step D3.2, list the nodes below the threshold [ Adaptively expand the number of virtual nodes according to the expansion function. Calculate the number of new virtual nodes for each node in the list of nodes below the threshold;

[0092] Step D3.3: After a new sharding computation request arrives, the new data is automatically reallocated to the list of nodes below the threshold. ]superior.

[0093] Compared with the prior art, the present invention has the following beneficial effects:

[0094] 1. This invention adopts a MapReduce task distribution and result aggregation mechanism to split the stock selection task into multiple sharded computing requests and distribute them to different nodes for parallel processing, effectively improving the real-time performance of the calculation.

[0095] 2. This invention achieves decoupling of computation and aggregation through an architecture that separates the front-end proxy service and the sharded computing service, thereby improving the system's concurrency capabilities, solving the problem that monolithic services cannot be scaled in parallel, and achieving efficient processing in high-concurrency scenarios.

[0096] 3. This invention introduces an adaptive consistent hashing algorithm to dynamically adjust the virtual nodes and hash ring, solving the problem of uneven sharding in traditional solutions when the number of stocks is small, and supporting elastic scaling of computing nodes.

[0097] 4. This invention adopts a pluggable indicator stock selection SDK design, which enables flexible replacement and rapid deployment of indicator calculation functions, ensuring the universality and stability of the overall system architecture.

[0098] 5. This invention obtains the corresponding stock pool based on sector factors as the benchmark stock pool list for stock selection. Through sharded calculation and adaptive consistent hash routing, the stock selection task is efficiently and evenly distributed to each sharded calculation node for parallel processing, which significantly improves the calculation speed and system throughput of large-scale indicator formula stock selection.

[0099] 6. The design concept of this invention is not only aimed at stock selection, but can also be horizontally extended to support similar scenarios that require real-time and efficient stock selection calculation, such as funds and futures.

[0100] 7. This invention introduces the concept of sector factors, which generally refers to a set of benchmark stocks for stock selection. This concept can be developed by the development team to achieve customized stock selection. Attached Figure Description

[0101] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0102] Figure 1 This is a diagram illustrating the overall architecture of the distributed high-efficiency index formula-based stock selection system in this embodiment of the invention.

[0103] Figure 2 This is a schematic diagram of the MapReduce task distribution and result aggregation mechanism of the distributed high-efficiency index formula stock selection system in an embodiment of the present invention;

[0104] Figure 3 This is a design diagram of the core module of the distributed high-efficiency index formula stock selection system in this embodiment of the invention;

[0105] Figure 4This is a flowchart illustrating the overall request process of the distributed high-efficiency indicator formula-based stock selection system in this embodiment of the invention. Detailed Implementation

[0106] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0107] This invention discloses a system for stock selection based on distributed, efficient indicator formulas, including a front-end agent service module, multiple sharding calculation service modules, and a data source service module. The front-end agent service module receives user requests containing sector factors and indicator formulas. Based on the sector factors, it obtains a list of benchmark stocks for selection and divides this list into multiple shards using the modulo operation of the incremental internal code and the total number of shards, generating corresponding sharding calculation requests. Its internal sharding calculation node management module distributes the requests to the corresponding sharding calculation service modules based on the shard identifiers using an adaptive consistent hashing algorithm. The sharding calculation service modules obtain stock information parameters based on the shard identifiers, call the embedded indicator stock selection SDK to execute indicator formula calculations, and return the filtering results. The system effectively improves the real-time performance and concurrent processing capabilities of stock selection calculations by using a MapReduce task distribution and result aggregation mechanism combined with an architecture that separates front-end proxy and sharded computation. It adopts an adaptive consistent hashing strategy to dynamically adjust the distribution of virtual and physical nodes and supports elastic scaling of nodes, solving the problems of uneven sharding and low efficiency in traditional solutions when the stock pool size changes. Ultimately, it achieves efficient and fast stock selection through a self-developed distributed solution. The indicator-based stock selection SDK adopts a pluggable design and supports user-defined formulas, enhancing flexibility and scalability while ensuring high system performance.

[0108] Example 1:

[0109] Figure 1 This is a diagram illustrating the overall architecture of the distributed high-efficiency index formula-based stock selection system in this embodiment of the invention.

[0110] like Figure 1 As shown, this embodiment provides a system for stock selection based on distributed high-efficiency indicator formulas, including: a front-end agent service module, multiple sharded computing service modules, and a data source service module;

[0111] The front-end proxy service module receives user requests, which include the sector factors and indicator formulas to be calculated. Sector factors broadly refer to various stock sectors, such as industry sectors or user-defined sectors, such as a watchlist of stocks. The AND / OR relationships between multiple sectors can also be used as a block factor; by default, it includes all A-shares in the market. The front-end proxy service module obtains the corresponding stock pool list based on the sector factors, serving as the benchmark stock pool list for stock selection. This can be achieved through the internally integrated sector SDK via API, RPC, or database queries.

[0112] Figure 2 This is a schematic diagram of the MapReduce task distribution and result aggregation mechanism of the distributed high-efficiency index formula stock selection system in this embodiment of the invention. Figure 3 This is a design diagram of the core module of the distributed high-efficiency indicator formula stock selection system in this embodiment of the invention. Figure 4 This is a flowchart illustrating the overall request process of the distributed high-efficiency indicator formula-based stock selection system in this embodiment of the invention.

[0113] like Figure 2 , 3 As shown in Figure 4, the front-end proxy service module adopts a MapReduce task distribution and result aggregation mechanism. It traverses the list of benchmark stock pools that need to be selected, performs a modulo operation based on the incremental internal code of each stock and the total number of shards to determine the shard to which each stock belongs, and generates multiple shard calculation requests. The total number of shards is a configurable parameter used to balance shard granularity and computational load. The total number of shards refers to the total number of shards that the front-end proxy service requests need to be divided into. The default value is 100. If it is too large, too many computational shard requests will be forwarded; if it is too small, too many stocks will need to be processed for each shard request. Each shard calculation request contains a list of sub-stock pools corresponding to a shard and a shard identifier. The front-end proxy service module, through its internal shard calculation node management module, distributes multiple shard calculation requests to the corresponding shard calculation service modules based on the shard identifier using an adaptive consistent hashing method.

[0114] Specifically, the process by which the front-end proxy service module generates multiple shard computation requests includes:

[0115] Step a1: The front-end agent service module internally maintains an incremental internal code mapping table for the entire A-share pool. The incremental internal code mapping table stores the correspondence between stock codes and incremental internal codes.

[0116] Step a2: The front-end agent service module subscribes to the change notifications of the entire market A-share pool list and dynamically maintains the incremental internal codes in the incremental internal code mapping table according to the stock code sorting. When the stock pool changes, the incremental internal codes are recalculated.

[0117] Step a3: The front-end proxy service module traverses the list of benchmark stock pools that need to be selected. For each stock in the list of benchmark stock pools that need to be selected, it performs a modulo operation based on the incremental internal code of the stock and the total number of shards to determine the shard identifier to which the stock belongs.

[0118] Step a4: The front-end proxy service module iterates through all shard identifiers, summarizes the stock list belonging to the same shard identifier, and forms a sub-stock pool list corresponding to each shard identifier. Modulo operations are used to ensure a relatively balanced number of stocks across shards.

[0119] Furthermore, the process by which the front-end proxy service module distributes multiple shard computing requests based on the shard identifier using an adaptive consistent hashing method, through its internal shard computing node management module, includes:

[0120] Step b1: The front-end proxy service module iterates through all shard identifiers and constructs a shard calculation request for each shard identifier. The shard calculation request includes the core fields of the user request, the shard identifier, and the corresponding sub-stock pool list.

[0121] In step b2, the front-end proxy service module concurrently submits multiple sharding computation requests to the sharding computation node management module through an internally maintained thread pool.

[0122] Step b3: The sharded compute node management module performs routing forwarding for each sharded compute request through its internally maintained scheduler. Routing forwarding includes:

[0123] Step b3.1: The scheduler maintains a hash ring, which maps all sharded computing service nodes and their associated virtual nodes. The scheduler uses a hash function to calculate the network addresses of all sharded computing service nodes and the identifiers of their associated virtual nodes, and maps the calculation results onto the hash ring.

[0124] In this embodiment, the MurmurHash hash function is used. This function is an open-source hash function released by Google, characterized by high performance, low collision rate, and uniform distribution. It uses a 32-bit function for processing to obtain... A hash ring of varying sizes.

[0125] Step b3.2: For each received sharding computation request, the scheduler calculates a hash value based on the shard identifier in the sharding computation request, and determines the target sharding computation service node on the hash ring based on the hash value.

[0126] The scheduler determines the target shard computation service node on the hash ring based on the hash value as follows: it queries the hash ring for the node closest to the hash value, which is either a physical node or a physical node to which a virtual node belongs. The hash function used by the scheduler to calculate the hash value based on the shard identifier is the same as the hash function used when constructing the hash ring.

[0127] In step b3.3, the scheduler forwards the sharding computation request to the target sharding computation service node.

[0128] Furthermore, the sharded computing node management module dynamically adjusts the number of physical nodes or the distribution of virtual nodes participating in the hash ring based on the real-time load status of each sharded computing service node using an adaptive consistent hashing method. The adaptive consistent hashing method includes an adaptive consistent hashing expansion node algorithm and an adaptive consistent hashing shrink node algorithm.

[0129] The adaptive consistent hashing algorithm dynamically scales up nodes by adding new physical sharding service nodes to the hash ring or reducing the number of virtual nodes attached to overloaded nodes when the node load exceeds a preset threshold. It also dynamically scales down nodes by removing the lowest-loaded physical sharding service node from the hash ring or adding more virtual nodes to low-loaded nodes when the node load falls below a preset threshold. By dynamically maintaining the hash ring, the adaptive consistent hashing algorithm automatically adjusts the distribution of physical and virtual nodes based on the real-time load of the sharding nodes, improving resource utilization and load balancing.

[0130] Furthermore, the adaptive consistent hashing resizing node algorithm is as follows:

[0131] DCHEN(S(t),Θ):

[0132]

[0133] in,

[0134] DCHEN is the adaptive consistent hashing node expansion processing function;

[0135] S(t) = A set of real-time node states;

[0136] This represents the current CPU utilization of the i-th shard computing service node;

[0137] This represents the current Mem memory usage of the i-th shard computing service node;

[0138] Θ={ , , ,start,end, , Configure parameters for the service system;

[0139] This is the upper limit threshold for Mem memory;

[0140] This is the CPU upper limit threshold;

[0141] The default number of virtual nodes is added when expanding the number of physical nodes;

[0142] The adjustment step size is the volatility that changes when the virtual node is scaled down.

[0143] `start` is the start time for the DCHEN function to execute; the default is the opening time.

[0144] `end` is the termination time for the DCHEN function, with the default closing time.

[0145] The maximum number of service nodes for a sharded computing node cluster;

[0146] This is a function to expand entity nodes for global overload.

[0147] This is a function for shrinking virtual nodes to handle single-point overload.

[0148] To record the relevant status data of each node in the disk;

[0149] The adaptive node expansion factor is defined as:

[0150]

[0151] The calculation method is to iterate through all nodes and calculate the percentage of CPU or Mem that exceeds the threshold.

[0152] in,

[0153] The adaptive consistent hashing algorithm for expanding nodes includes the following steps:

[0154] In step C1, the sharded computing node management module within the front-end agent service module asynchronously executes the DCHEN(S(t),Θ) ​​function based on the system parameters start and end times, and performs real-time calculations based on S(t). The value is used to perform timed judgment and processing of the expansion process; in order to ensure that the service status of the computing nodes is accurately reflected during peak periods, the status data of each node is collected at the default disk minute level, and the non-disk time is not meaningful because the request volume is reduced.

[0155] Step C2, when ≥0.5 ,trigger Expand node operation.

[0156] Furthermore, in step C2 The node expansion operation includes the following sub-steps:

[0157] Step C2.1: The sharded compute node management module inside the front-end agent service module notifies the node operation through the service bus, performs the expansion operation through the K8S API, and completes the registration of the new sharded compute node service.

[0158] Step C2.2, New entity node After automatic expansion, the entity nodes will be... of Number of virtual nodes and entity nodes It is added to the hash ring itself;

[0159] Step C2.3: Upon the arrival of a new sharding computation request, entity nodes are automatically reallocated. and virtual nodes Surrounding sharded data has not been assigned to entity nodes. and virtual nodes The fragmented data continues to fall to the previous node.

[0160] The above operations complete the relevant expansion of entity nodes.

[0161] Step C3, when When triggered Perform a shrinkage operation on the specified virtual node.

[0162] Furthermore, in step C3 Performing a virtual node scaling-down operation on a specified node includes the following sub-steps:

[0163] Step C3.1: The sharding node management module within the front-end agent service module iterates through the list of all nodes exceeding the threshold. ];

[0164] Step C3.2, list the nodes that exceed the threshold [ Adaptively reduce the number of virtual nodes according to the shrinkage function. Calculate the number of new virtual nodes for each node in the list of nodes that exceed the threshold;

[0165] Step C3.3: Upon the arrival of a new sharding computation request, the list of nodes exceeding the threshold is automatically reallocated. The data fragments are limited to the overloaded nodes and their scaling down. On the scaling virtual node, other shard data remains unaffected. The above operations complete the relevant scaling virtual node operation.

[0166] Step C4: If the conditions of steps C2 and C3 are not met, execute... The function performs a node state storage procedure, recording the state of each shard computing node on disk according to time points, providing a data source for subsequent scaling-down node analysis and processing.

[0167] Furthermore, the adaptive consistent hashing shrinking node algorithm is as follows:

[0168] DCHSN( ,Θ):

[0169]

[0170] in,

[0171] DCHSN is the adaptive consistent hash shrinking node processing function;

[0172] This is a collection of the states of each node during the latest transaction period;

[0173] ={S(t)|t∈{ , ..., }};

[0174] S(t) = / / A set of node states at a specified time point;

[0175] Θ={ , , , ,α,β,start,end,collectTime} are the service system configuration parameters; Minimum number of service nodes for a sharded computing node cluster; The fluctuation rate when expanding the capacity of virtual nodes is the adjustment step size. This is the lower limit threshold for CPU usage. α is the lower limit threshold for Mem memory usage; α is the CPU computation ratio during scaling down computation; β is the Mem computation ratio during scaling down computation; start is... The start time of the statistical node status set is the default opening time; end is the end time of the statistical node status set is the default closing time; collectTime is the time point for the DCHSN function to process the data.

[0176] The adaptive node shrinkage factor is defined as follows:

[0177]

[0178] The calculation method involves iterating through all nodes and calculating the percentage of CPU usage for each node whose Mem is below a threshold.

[0179] This is a function for globally low-load shrinking entity nodes;

[0180] Functions for expanding virtual nodes with low load at a single point;

[0181] The adaptive consistent hashing shrinking node algorithm includes the following steps:

[0182] Step D1: The sharded computing node management module within the front-end agent service module asynchronously executes DCHSN based on the system parameter collectTime. The function ,Θ) is based on Calculate the day The value is used to perform timed judgment and processing of the scaling down process. In order to reduce the impact of intraday requests, the timed execution is carried out once every night of the trading day, such as 23:00.

[0183] Step D2, when the global low-load condition is met ( When triggered Reduce node operations.

[0184] Specifically, in step D2 The node reduction operation includes the following sub-steps:

[0185] In step D2.1, when the sharded computing node management module inside the front-end agent service module executes the node reduction function, it iterates through... The average state value of each node is calculated and the node with the smallest percentage is identified. ,calculate:

[0186] = ,

[0187] in,

[0188] This represents the CPU utilization of the j-th shard computing service node at time t;

[0189] This represents the memory usage of Mem at time t for the j-th shard computing service node;

[0190] for The total number of time points collected in the collection is used to calculate the target node with the lowest average CPU and Mem usage based on this formula. This node is then selected as the node to be cleaned up during the scaling-down process.

[0191] Step D2.2: The internal node management module of the front-end agent service module notifies the node cleanup operation through the service bus and performs node shrinking operation through the K8S API.

[0192] Step D2.3, After the node is successfully cleaned up, the node is triggered. Related virtual nodes and nodes It is removed from the hash ring;

[0193] Step D2.4: Upon receiving a new sharding computation request, nodes are automatically reallocated. and virtual nodes The surrounding sharded data is transferred to other nodes for processing.

[0194] The above operations complete the relevant shrinking entity node operation.

[0195] Step D3, when the single-node low-load condition is met When triggered Perform virtual node expansion operations on the specified node.

[0196] When neither the full load nor the single node load conditions are met, it indicates that the current state is relatively balanced and stable and no action is required.

[0197] Specifically, in step D3 Performing a virtual node expansion operation for a specified node includes the following sub-steps:

[0198] Step D3.1: The sharding computing node management module within the front-end proxy service module finds all the sharded computing nodes by traversing the process. List of nodes below the threshold [ ].

[0199] Step D3.2, list the nodes below the threshold [ Adaptively expand the number of virtual nodes according to the expansion function. Calculate the number of new virtual nodes for each node in the list of nodes below the threshold.

[0200] Step D3.3: After a new sharding computation request arrives, the new data is automatically reallocated to the list of nodes below the threshold. The impact is limited to the expansion of low-load nodes. On the proportional virtual node, other shard data remains unaffected. The above operations complete the relevant virtual node expansion operation.

[0201] The sharding calculation service module receives sharding calculation requests. Based on the sharding identifier in the sharding calculation request, the sharding calculation service module obtains the corresponding stock information parameters from the data source service module. The sharding calculation service module calls the embedded indicator stock selection SDK module, inputs the stock information parameters into the preset indicator formula calculation function for processing, obtains the stock screening results, and returns them to the front-end agent service module.

[0202] Specifically, the indicator-based stock selection SDK module is accessed through a predefined application programming interface (API), adopts a pluggable design, and supports user-defined formulas, thereby enhancing flexibility and scalability while ensuring high system performance.

[0203] The data source service module is used to provide stock information parameters to the sharding calculation service module.

[0204] The front-end agent service module aggregates the stock screening results returned by multiple sharded computing service modules to generate the final stock selection results.

[0205] Specifically, after concurrently distributing multiple sharding calculation requests, the front-end agent service module blocks and waits for the stock screening results corresponding to all sharding calculation requests to be returned, and then summarizes all the returned stock screening results to generate the final stock selection result.

[0206] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0207] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A stock selection system based on distributed high-efficiency indicator formulas, characterized in that, include: The module includes a front-end agent service module, multiple sharded computing service modules, and a data source service module. The front-end proxy service module receives user requests, which include sector factors and indicator formulas to be calculated. The front-end proxy service module obtains a corresponding stock pool list based on the sector factors as a benchmark stock pool list for stock selection. The front-end proxy service module iterates through the benchmark stock pool list, performs a modulo operation based on the incremental internal code of each stock and the total number of shards to determine the shard to which each stock belongs, and generates multiple shard calculation requests. Each shard calculation request includes a sub-stock pool list corresponding to a shard and a shard identifier. The front-end proxy service module, through its internal shard calculation node management module, distributes the multiple shard calculation requests to the corresponding shard calculation service modules based on the shard identifier using an adaptive consistent hashing method. The process by which the front-end proxy service module generates multiple sharding computation requests includes: Step a1: The front-end agent service module internally maintains an incremental internal code mapping table for the entire A-share pool, and the incremental internal code mapping table stores the correspondence between stock codes and incremental internal codes; Step a2: The front-end agent service module subscribes to the change notification of the entire market A-share pool list and dynamically maintains the incremental internal code in the incremental internal code mapping table according to the stock code sorting. Step a3: The front-end proxy service module traverses the list of benchmark stock pools that need to be selected, and for each stock in the list of benchmark stock pools that need to be selected, performs a modulo operation based on the incremental internal code of the stock and the total number of shards to determine the shard identifier to which the stock belongs. Step a4: The front-end agent service module traverses all shard identifiers, summarizes the stock list belonging to the same shard identifier, and forms the sub-stock pool list corresponding to each shard identifier. The sharding calculation service module is used to receive the sharding calculation request; the sharding calculation service module obtains the corresponding stock information parameters from the data source service module according to the sharding identifier in the sharding calculation request; the sharding calculation service module calls the embedded indicator stock selection SDK module, inputs the stock information parameters into the indicator formula for processing, obtains the stock screening result and returns it to the front-end agent service module; the indicator stock selection SDK module is accessed through a predefined application programming interface. The data source service module is used to provide the stock information parameters to the sharding calculation service module; The front-end agent service module aggregates the stock screening results returned by the multiple sharded computing service modules to generate the final stock selection results; The process by which the front-end proxy service module distributes multiple sharding computation requests using an adaptive consistent hashing method based on the sharding identifier through its internal sharding computation node management module includes: Step b1: The front-end proxy service module traverses all shard identifiers and constructs a shard calculation request for each shard identifier. The shard calculation request includes the core fields of the user request, the shard identifier, and the corresponding sub-stock pool list. Step b2: The front-end proxy service module concurrently submits multiple sharding computation requests to the sharding computation node management module through an internally maintained thread pool. Step b3, the sharded computing node management module performs routing forwarding for each sharded computing request through an internally maintained scheduler. The routing forwarding includes: Step b3.1, the scheduler maintains a hash ring, on which all sharded computing service nodes and their associated virtual nodes are mapped; Step b3.2: For each received sharding computation request, the scheduler calculates a hash value based on the sharding identifier in the sharding computation request, and determines the target sharding computation service node on the hash ring based on the hash value; Step b3.3: The scheduler forwards the sharding computation request to the target sharding computation service node; The sharded computing node management module dynamically adjusts the number of physical nodes or the distribution of virtual nodes participating in the hash ring based on the real-time load status of each sharded computing service node using an adaptive consistent hashing method. The adaptive consistent hashing method includes an adaptive consistent hashing expansion node algorithm and an adaptive consistent hashing shrink node algorithm.

2. The system for stock selection based on distributed high-efficiency indicator formulas according to claim 1, characterized in that, The adaptive consistent hashing expansion node algorithm is as follows: DCHEN(S(t),Θ): in, DCHEN is the adaptive consistent hashing node expansion processing function; S(t) = A set of real-time node states; This represents the current CPU utilization of the i-th shard computing service node; This represents the current Mem memory usage of the i-th shard computing service node; Θ={ , , ,start,end, , Configure parameters for the service system; This is the upper limit threshold for Mem memory; This is the CPU upper limit threshold; The default number of virtual nodes is added when expanding the number of physical nodes; The adjustment step size is the volatility that changes when the virtual node is scaled down. `start` is the start time for the DCHEN function to execute; the default is the opening time. `end` is the termination time for the DCHEN function, with the default closing time. The maximum number of service nodes for a sharded computing node cluster; This is a function to expand entity nodes for global overload. This is a function for shrinking virtual nodes to handle single-point overload. To record the relevant status data of each node in the disk; The adaptive node expansion factor is defined as: The calculation method is to iterate through all nodes and calculate the percentage of CPU or Mem that exceeds the threshold. in, The adaptive consistent hashing node expansion algorithm includes the following steps: Step C1: The sharded computing node management module inside the front-end agent service module asynchronously executes the DCHEN(S(t),Θ) ​​function according to the system parameters start and end times, and calculates in real time based on S(t). The value is used to perform timed judgment and processing of the expansion process; Step C2, when ≥0.5 ,trigger Expand node operation; Step C3, when When triggered Perform a shrinkage operation on the specified virtual node; Step C4: If the conditions of steps C2 and C3 are not met, execute... The function performs a node state storage procedure, recording the state of each shard computing node to disk according to time points.

3. The system for stock selection based on distributed high-efficiency indicator formulas according to claim 2, characterized in that, In step C2 The node expansion operation includes the following sub-steps: Step C2.1: The sharded compute node management module inside the front-end agent service module notifies the node operation through the service bus, performs the expansion operation through the K8S API, and completes the registration of the new sharded compute node service. Step C2.2, New entity node After automatic expansion, the entity nodes will be... of Number of virtual nodes and entity nodes It is added to the hash ring itself; Step C2.3: Upon the arrival of a new sharding computation request, entity nodes are automatically reallocated. and virtual nodes Surrounding sharded data has not been assigned to entity nodes. and virtual nodes The fragmented data continues to fall to the previous node.

4. The system for stock selection based on distributed high-efficiency indicator formulas according to claim 2, characterized in that, In step C3 Performing a virtual node scaling-down operation on a specified node includes the following sub-steps: Step C3.1: The sharding node management module within the front-end agent service module iterates through the list of all nodes exceeding the threshold. ]; Step C3.2, for the list of nodes exceeding the threshold [ Adaptively reduce the number of virtual nodes according to the shrinkage function. Calculate the number of new virtual nodes for each node in the list of nodes that exceed the threshold; Step C3.3: Upon the arrival of a new sharding computation request, the list of nodes exceeding the threshold is automatically reassigned. The fragmented data.

5. The system for stock selection based on distributed high-efficiency indicator formulas according to claim 1, characterized in that, The adaptive consistent hash shrinking node algorithm is as follows: DCHSN( ,Θ): in, DCHSN is the adaptive consistent hash shrinking node processing function; This is a collection of the states of each node during the latest transaction period; ={S(t)|t∈{ , ..., }}; S(t) = / / A set of node states at a specified time point; Θ={ , , , ,α,β,start,end,collectTime} are the service system configuration parameters; Minimum number of service nodes for a sharded computing node cluster; The fluctuation rate when expanding the capacity of virtual nodes is the adjustment step size. This is the lower limit threshold for CPU usage. α is the lower limit threshold for Mem memory usage; α is the CPU computation ratio during scaling down computation; β is the Mem computation ratio during scaling down computation; start is... The start time of the statistical node status set is the default opening time; end is the end time of the statistical node status set is the default closing time; collectTime is the time point for the DCHSN function to process the data. The adaptive node shrinkage factor is defined as follows: The calculation method involves iterating through all nodes and calculating the percentage of CPU usage for each node whose Mem is below a threshold. This is a function for globally low-load shrinking entity nodes; Functions for expanding virtual nodes with low load at a single point; The adaptive consistent hashing shrinking node algorithm includes the following steps: Step D1: The sharded computing node management module inside the front-end agent service module asynchronously executes DCHSN (based on the system parameter collectTime) on a timed basis. The function ,Θ) is based on Calculate the day The value is used to perform timed judgment and processing of the scaling down process; Step D2, when When triggered Reduce node operations; Step D3, when When triggered Perform virtual node expansion operations on the specified node.

6. The system for stock selection based on distributed high-efficiency indicator formulas according to claim 5, characterized in that, In step D2 The node reduction operation includes the following sub-steps: Step D2.1: When the sharded computing node management module inside the front-end proxy service module executes the node reduction function, it traverses... The average state value of each node is calculated and the node with the smallest percentage is identified. ,calculate: = ， in, This represents the CPU utilization of the j-th shard computing service node at time t; This represents the memory usage of Mem at time t for the j-th shard computing service node; for The total number of time points collected in the set is used to calculate the target node with the lowest average CPU and Mem utilization based on this formula. Step D2.2: The internal node management module of the front-end agent service module notifies the node cleanup operation through the service bus and performs node shrinking operation through the K8S API. Step D2.3, After the node is successfully cleaned up, the node is triggered. Related virtual nodes and nodes It is removed from the hash ring; Step D2.4: Upon receiving a new sharding computation request, nodes are automatically reallocated. and virtual nodes The surrounding sharded data is transferred to other nodes for processing.

7. The system for stock selection based on distributed high-efficiency indicator formulas according to claim 6, characterized in that, In step D3 Performing a virtual node expansion operation for a specified node includes the following sub-steps: Step D3.1, the sharded computing node management module inside the front-end proxy service module finds all the sharded computing nodes by traversing the network. List of nodes below the threshold [ ]; Step D3.2, for the list of nodes below the threshold [ Adaptively expand the number of virtual nodes according to the expansion function. Calculate the number of new virtual nodes for each node in the list of nodes below the threshold; Step D3.3: After a new sharding computation request arrives, the new data is automatically reallocated to the list of nodes below the threshold. ]superior.

Images