Index processing scheduling method and system based on distributed calculation, and storage medium

Abstract

The invention discloses a distributed calculation index processing scheduling method and system and a storage medium. The method comprises the following steps: receiving a registration request of a computing node, and periodically synchronizing index configuration and rule group information from a synchronization management center; acquiring a processing request sent by a user; analyzing the processing request, matching an index group according to the index configuration and the rule group information, packaging index information corresponding to the index group, and constructing a directed acyclic graph; obtaining a to-be-processed index by using the directed acyclic graph, and generating an index processing distribution task; sending the index processing distribution task to a corresponding computing node, so that the computing node performs index processing, and feeding back a processing result; and receiving a processing result, and processing the processing result. By implementing the method provided by the invention, the index processing process can be optimized, the calculation efficiency, flexibility and real-time performance can be improved, dynamic selective calculation and resource optimization allocation can be realized, and the maintenance cost can be reduced, so that complex service requirements and efficient processing in a big data scene can be better supported.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically to an indicator processing scheduling method, system and storage medium based on distributed computing. Background Art

[0002] The indicator processing system uses efficient data processing and calculation logic to flexibly process and analyze a large number of business indicators, improving decision-making support capabilities, optimizing resource utilization, and enhancing the system's real-time and scalability. The current indicator processing system mainly relies on static configuration files to store indicator processing logic. When a task is triggered, the system adopts a full-module calculation mode, which cannot achieve incremental or selective calculation of some indicators. In big data scenarios, the system can only increase concurrent processing capabilities and throughput by horizontally expanding multiple computing node replicas. However, this design approach has obvious limitations, especially when faced with complex business needs. It cannot flexibly select and process specific indicators, which affects the overall efficiency and adaptability of the system.

[0003] The main defects of existing technologies include: first, the full-module computing mode leads to low computing efficiency, making selective computing impossible and resulting in resource waste; second, there is a lack of computing flexibility at the indicator level, making it impossible to dynamically select indicators for calculation based on specific business needs; at the same time, resource utilization is also uneven, and computing resources may be overly concentrated on some complex indicators, resulting in unreasonable overall resource allocation; in addition, existing systems have difficulty supporting real-time or near-real-time computing, especially when tasks are concentrated, and cannot meet efficient computing needs; finally, because the processing logic is stored in static configuration files, the maintenance and update costs are high, and updating the logic requires reloading the configuration or restarting the system. These problems make it difficult for existing designs to meet the performance, flexibility, and real-time requirements of modern data processing.

[0004] Therefore, it is necessary to design a new method to optimize the indicator processing process, improve computing efficiency, flexibility and real-time performance, realize dynamic selective computing, optimize resource allocation and reduce maintenance costs, so as to better support complex business needs and efficient processing in big data scenarios. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide an indicator processing scheduling method, system and storage medium based on distributed computing.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an indicator processing scheduling method based on distributed computing, comprising:

[0007] Receive registration requests from computing nodes and regularly synchronize indicator configuration and rule group information from the synchronization management center;

[0008] Get the processing request sent by the user;

[0009] Parsing the processing request, matching the indicator group according to the indicator configuration and rule group information, encapsulating the indicator information corresponding to the indicator group, and constructing a directed acyclic graph of the dependency relationship;

[0010] Utilizing the directed acyclic graph to obtain indicators to be processed, and generating indicator processing and distribution tasks;

[0011] Sending the indicator processing distribution task to the corresponding computing node, so that the computing node performs indicator processing and feeds back the processing result;

[0012] The processing results are received and cached in a temporary storage. A third-party service persists the processing results to ElasticSearch, notifies the user of the corresponding processing results through a message queue, and returns the processing results when the user queries.

[0013] Its further technical solution is: the indicator configuration includes English name, Chinese name, data source, screening condition, expression, parent indicator; the rule group information includes information corresponding to a set of multiple indicators.

[0014] A further technical solution is: parsing the processing request, matching the indicator group according to the indicator configuration and rule group information, encapsulating the indicator information corresponding to the indicator group, and constructing a directed acyclic graph of the dependency relationship, including:

[0015] Parsing the processing request to extract request rules and related user information;

[0016] Searching for an indicator group that meets the requirements from the indicator configuration and rule group information according to the request rule;

[0017] Extracting and encapsulating corresponding metadata for each indicator in the indicator group to obtain a structured data packet;

[0018] The dependency relationship of each indicator in the indicator group is parsed, a dependency graph between the indicators is constructed, each indicator in the indicator group is traversed, the indicator is added as a node, and dependency edges are established to obtain a directed acyclic graph of the dependency relationship.

[0019] A further technical solution is: parsing the dependency relationship of each indicator in the indicator group, constructing a dependency graph between the indicators, traversing each indicator in the indicator group, adding the indicator as a node, and establishing dependency edges to obtain a directed acyclic graph of the dependency relationship, including:

[0020] Analyze the dependency relationship of each indicator in the indicator group, use the DAG package in Python to build a dependency graph between the indicators, traverse each indicator in the indicator group, use the add_node() method to add the indicator as a node, and use the add_edge() method to establish a dependency edge to obtain a directed acyclic graph of the dependency relationship.

[0021] A further technical solution is: using the directed acyclic graph to obtain the indicators to be processed and generating indicator processing and distribution tasks, including:

[0022] Obtain all leaf nodes that do not depend on other indicators from the directed acyclic graph through the all_leaves() method of DAG to obtain the indicators to be processed;

[0023] The indicators to be processed are packaged to generate indicator processing and distribution tasks.

[0024] A further technical solution is that the step of sending the indicator processing and distribution task to the corresponding computing node so that the computing node processes the indicator and feeds back the processing result includes:

[0025] Send the indicator processing and distribution task to the corresponding computing node, so that the computing node puts the indicator processing and distribution task into the queue to be processed, and uses the Pandas library and predefined calculation methods to process the tasks in the queue to be processed. During the processing, the computing node calculates the task according to the expression and screening conditions of the indicator. If the processing is successful, the status of the indicator is updated to success, and the processing result is fed back; if the processing fails, the status of the indicator is updated to failure, and the processing result is fed back; wherein, when the computing node completes the processing of the indicator, it will encapsulate the processing result into a data packet and return it through the UDP protocol.

[0026] A further technical solution is: receiving the processing result, caching the processing result in a temporary storage, having a third-party service persist the processing result to ElasticSearch, notifying the user of the corresponding processing result through a message queue, and returning the processing result when the user queries, including:

[0027] Receive the processing results, process the status of the indicators according to the dependency relationship of the directed acyclic graph, cache the processing results in a temporary storage, and have a third-party service persist the processing results to ElasticSearch, notify the user of the corresponding processing results through a message queue, and return the processing results when the user queries.

[0028] The present invention also provides an indicator processing scheduling system based on distributed computing, comprising:

[0029] Periodic update unit, used to receive registration requests from computing nodes and regularly synchronize indicator configuration and rule group information from the synchronization management center;

[0030] A request acquisition unit, used to acquire a processing request sent by a user;

[0031] a parsing unit, configured to parse the processing request, match an indicator group according to the indicator configuration and rule group information, encapsulate indicator information corresponding to the indicator group, and construct a directed acyclic graph of dependency relationships;

[0032] an acquisition unit, configured to acquire the indicators to be processed using the directed acyclic graph and generate indicator processing and distribution tasks;

[0033] a sending unit, configured to send the indicator processing distribution task to the corresponding computing node, so that the computing node performs indicator processing and feeds back the processing result;

[0034] The receiving unit is used to receive the processing results and cache the processing results in a temporary storage. The third-party service persists the processing results to ElasticSearch, notifies the user of the corresponding processing results through the message queue, and returns the processing results when the user queries.

[0035] Its further technical solution is: the analysis unit includes:

[0036] an extraction subunit, configured to parse the processing request to extract request rules and related user information;

[0037] A search subunit, configured to search for an indicator group that meets the requirements from the indicator configuration and rule group information according to the request rule;

[0038] The encapsulation subunit is used to extract and encapsulate corresponding metadata for each indicator in the indicator group to obtain a structured data packet;

[0039] The graph construction subunit is used to parse the dependency relationship of each indicator in the indicator group, construct a dependency graph between the indicators, traverse each indicator in the indicator group, add the indicator as a node, and establish dependency edges to obtain a directed acyclic graph of the dependency relationship.

[0040] The present invention further provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0041] The beneficial effects of the present invention compared with the existing technology are as follows: the present invention realizes flexible indicator configuration and rule matching by receiving computing node registration requests and regularly synchronizing configuration and rule group information; it optimizes the dependency of indicators by parsing user processing requests and constructing a directed acyclic graph, thereby generating efficient indicator processing tasks and distributing the tasks to computing nodes for execution; the processing results are temporarily cached and persisted to ElasticSearch by a third-party service, and the user is notified and the results are returned in time through the message queue; the design of this process not only optimizes the indicator processing process, improves computing efficiency and real-time performance, but also supports dynamic selective computing and resource optimization allocation, thereby reducing maintenance costs, and can flexibly respond to complex business needs and efficient processing in big data scenarios.

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of an application scenario of the index processing scheduling method based on distributed computing provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a flow chart of an indicator processing scheduling method based on distributed computing provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a sub-process of an indicator processing scheduling method based on distributed computing provided by an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of a sub-process of an indicator processing scheduling method based on distributed computing provided by an embodiment of the present invention;

[0048] Figure 5 A schematic block diagram of an indicator processing scheduling system based on distributed computing provided by an embodiment of the present invention;

[0049] Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0052] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0053] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0054] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the index processing scheduling method based on distributed computing provided in an embodiment of the present invention. Figure 2 The schematic flow chart of the index processing scheduling method based on distributed computing provided by the embodiment of the present invention is applied to a server. Figure 1The main service in the system is the MASTER service. The configuration management center, namely the MGR service, is responsible for storing and managing the indicator configuration and rule groups in the system; providing the latest configuration information to the master service (MASTER service); the data source is the configuration database; each indicator contains the following key information: English name, Chinese name, data source, filter conditions, calculation expression and parent indicator.; The indicator group is a collection of multiple indicators; the request rule is bound to the indicator group. The MASTER service regularly pulls the latest configuration information from the configuration management center; accepts requests from API or message queue (MQ), and the message body contains information such as the user's unique identifier, request rules and topic name; after packaging the data, it distributes the computing tasks to multiple computing nodes (NODE services) through the UDP protocol; receives the results returned by the computing nodes, summarizes them, and writes the final results to Redis; implements dynamic task scheduling, load balancing and fault tolerance mechanisms to ensure the stability and reliability of the system. The computing node, namely the NODE service, develops and predefines calculation methods, such as group aggregation methods. An example calculation expression is: group_by_a_and_sum_b(data = pd_and_filter(data = data, filters = filters), a = a, b = b). During system initialization, predefined calculation methods (such as the data filtering method pd_and_filter and the grouped statistics method group_by_a_and_sum_b) are loaded into memory. The pd_and_filter method is used to filter data, while the group_by_a_and_sum_b method performs grouping and summation calculations, returning customized statistical results. The system receives processing tasks from the master control service, converts the indicator expression string into a valid expression using the eval method, and then calls the predefined calculation method for processing. After calculation, the result is packaged and returned to the master control service via the UDP protocol. The data writing service (WORK service) monitors the calculation results in Redis and executes data write operations. A message queue callback notifies the business end of the completion of data processing. Redis serves as temporary storage, ensuring data reliability during transmission and processing. Cluster deployment is supported to improve storage performance and scalability. The Message Queue (MQ) is used to receive processing requests and issue callback notifications, enabling asynchronous communication and decoupling. ElasticSearch is used to store calculation results and provide efficient query capabilities.

[0055] The method of this embodiment optimizes the indicator processing process and improves computing efficiency, flexibility and real-time performance through an indicator processing scheduling mechanism based on distributed computing. First, by receiving the registration request of the computing node and the indicator configuration and rule group information of the synchronization management center, real-time updates and flexible adjustments are ensured. By constructing a directed acyclic graph (DAG) of dependencies, the indicators to be processed are dynamically selected to support the distribution of complex computing tasks and parallel processing. On the computing node, Pandas and predefined calculation methods are used to complete the efficient calculation of the task, ensuring real-time feedback on the success and failure of the task. The processing results are cached and persisted through ElasticSearch, and the user is notified in real time in combination with the message queue, thereby achieving efficient support for complex business needs in big data scenarios. This method also achieves optimal allocation of resources, reduces maintenance costs, and ensures the scalability and dynamic adaptability of the system.

[0056] Figure 2 This is a flow chart of the index processing scheduling method based on distributed computing provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S160.

[0057] S110: Receive a registration request from a computing node, and regularly synchronize indicator configuration and rule group information from a synchronization management center.

[0058] In this embodiment, the indicator configuration includes an English name, a Chinese name, a data source, a screening condition, an expression, and a parent indicator; the rule group information includes information corresponding to a set of multiple indicators.

[0059] In this embodiment, the computing node (i.e., NODE service) needs to register with the master service (MASTER service). To ensure that the entire system can correctly distribute computing tasks, the master service needs to know which computing nodes are online and can receive tasks. The registration request of the computing node includes the following:

[0060] Identification of computing nodes: Each computing node should have a unique identifier, usually an ID generated based on its hardware information or configuration.

[0061] Computing capabilities: A computing node may need to report its computing capabilities, such as processing power, storage capacity, etc., so that the master control service can reasonably allocate tasks to the node with the most suitable performance.

[0062] Communication capabilities: Compute nodes report their supported communication protocols and ports to the master control service, which then uses this information to send tasks to the compute nodes.

[0063] Compute node registration typically occurs during system initialization. Compute nodes communicate with the master control service over the network, passing registration information to the master control service. Upon receiving the registration request, the master control service stores the node information in the system, marks the node as registered, and prepares to dispatch tasks to the node in subsequent task scheduling.

[0064] The MGR service is responsible for storing and managing the system configuration, including indicator configuration and rule group information. To ensure that the master service can obtain the latest configuration information at any time, the master service needs to regularly send requests to the MGR service to synchronize the latest configuration information. This process mainly involves the following aspects:

[0065] Indicator configuration is the definition information of each indicator in the system, including the following:

[0066] English name: The English name of the indicator, used to identify the indicator.

[0067] Chinese name: The Chinese name of the indicator, which is easier for Chinese users to understand.

[0068] Data source: The source of the indicator, which may be a database, API interface, etc., is used to indicate where the indicator obtains data from.

[0069] Filter conditions: The conditions for filtering indicator data, which may be certain time ranges, data types, and other restrictions.

[0070] Expression: The expression used to calculate the indicator, such as aggregation, calculation formula, etc.

[0071] Parent indicator: If the indicator depends on other indicators (i.e., there is a parent-child relationship), the parent indicator will be marked to ensure the calculation order and dependency of the data.

[0072] By regularly synchronizing these indicator configurations, the master service ensures that it has the latest indicator definitions. The master service loads this information into memory, allowing it to schedule and distribute computing tasks at any time.

[0073] A rule group is a collection of multiple indicators. Each rule group represents a set of indicators and calculation rules used to uniformly process a type of data. Rule group information includes:

[0074] Rule group identifier: Each rule group has a unique identifier, which is used to distinguish different rule groups.

[0075] Included indicators: Each rule group contains multiple indicators. When the master service synchronizes the rule group, it obtains the IDs and related information of all indicators included in each rule group.

[0076] Calculation method of the rule group: Each rule group may have a different calculation method or processing logic. The master control service will update this information during synchronization.

[0077] By regularly synchronizing rule group information, the master control service can ensure that the indicators and calculation rules in the rule group are always up to date, avoiding task processing anomalies caused by configuration changes.

[0078] The Master Service periodically requests the latest configuration and rule group information from the Synchronization Manager at predetermined intervals. The frequency of synchronization cycles may be adjusted based on system requirements. To avoid burdening system performance with frequent synchronization requests, an appropriate interval (e.g., hourly, daily, etc.) is typically set.

[0079] Once the master service successfully obtains the new indicator configuration and rule group information from the synchronization management center, it will parse and process this data:

[0080] Load Configuration: Loads new indicator configuration information into memory and ensures that configuration changes take effect immediately. This means that when the indicator configuration or rule group in the MGR service changes, the master service automatically loads the latest configuration to ensure that subsequent calculations use the latest rules.

[0081] Update rule groups: If the rule group information changes, the master control service will update the internal rule group mapping relationship and ensure that the new rules are applied in subsequent computing tasks.

[0082] The master control service plays a core scheduling role in the system by synchronizing the indicator configuration and rule group information obtained from the management center:

[0083] It is responsible for obtaining indicator data from computing nodes and performing data processing and calculation.

[0084] It distributes tasks according to rule group information and manages the dependencies between tasks through a directed acyclic graph (DAG).

[0085] It also needs to return the results to Redis or other storage media based on the calculation results of the task for the business party to query and use.

[0086] S110's primary responsibility is to ensure that the master control service regularly synchronizes and updates indicator configurations and rule group information. This synchronization process ensures that the master control service always processes the latest indicators and calculation rules, ensuring that the system's computing tasks are executed according to the latest configuration. Furthermore, the master control service receives registration requests from compute nodes and ensures that tasks are assigned to the appropriate compute nodes, ultimately achieving efficient data processing.

[0087] Specifically, the configuration management service provides an independent thread periodic request to the master service synchronization interface; during the request process, the master service will provide the latest request rules, indicator groups, indicators, and data dependencies in full return.

[0088] The configuration management service receives data, parses the request rules, indicator groups, indicators, and data dependencies, and loads them into memory, ensuring that changes take effect immediately.

[0089] S120: Obtain a processing request sent by the user.

[0090] In this embodiment, the request sent by the user contains relevant information of the processing requirements, such as the user's unique identifier, request rules, subject name, etc. The user request is transmitted through the message queue (MQ), ensuring asynchronous processing of the request and system decoupling.

[0091] User unique identifier: This is an identifier used to identify the source of the request, ensuring that the user's request can be accurately tracked and the corresponding results are ultimately returned.

[0092] Request rules: This part refers to the parameters of the rules or indicator processing that the user wants to execute, including filtering conditions, calculation methods, etc.

[0093] Subject Name: This field identifies the type of request or specific business area for further processing and classification.

[0094] S130: Parse the processing request, match the indicator group according to the indicator configuration and rule group information, encapsulate the indicator information corresponding to the indicator group, and construct a directed acyclic graph of the dependency relationship.

[0095] In this embodiment, an indicator group is a collection of multiple indicators (usually a number of data processing units). In this system, indicator groups and rule groups are closely related. An indicator group can contain multiple indicators, and these indicators may have dependencies. Through configuration files or configuration services (such as the MGR service), the system can load these indicator groups and rule sets and use them for subsequent data processing and calculations.

[0096] Metric group: A collection of multiple metrics. Each metric represents a specific computing task.

[0097] Indicator configuration: Contains configuration information about each indicator, such as the indicator's English name, Chinese name, data source, filter conditions, calculation expression, and parent indicator (parent-child dependency).

[0098] Rule groups: These are typically related to business logic and specify how metrics are grouped or filtered. Rule groups are closely related to metric groups.

[0099] Each metric has a set of metadata information that describes the attributes and execution context of the metric. In the system, this metadata includes but is not limited to:

[0100] Unique identifier of the indicator: used to uniquely identify each indicator.

[0101] Calculation expression: describes how to process the indicator's data, such as aggregation operations, filter conditions, and expressions.

[0102] Filter conditions: Filter conditions for data, used to limit the scope or type of data.

[0103] Data source: The source of the input data for this indicator (such as database, file, streaming data, etc.).

[0104] Parent indicator: If the indicator depends on other indicators (parent indicators), their dependencies will be recorded.

[0105] After parsing the request and loading it from the configuration center, this information will be encapsulated into a structured data packet to form indicator information for subsequent data processing and calculation.

[0106] A directed acyclic graph (DAG) is used to represent the dependencies between metrics. Each metric is considered a node in the graph, and directed edges between nodes represent dependencies—that is, the calculation of one metric depends on the results of other metrics. DAGs ensure the sequential nature of computational tasks and avoid circular dependencies. Using DAGs, metrics can be calculated in the correct order during the computation process.

[0107] Node: Each metric in the graph is considered a node.

[0108] Edge: A directed edge represents a dependency relationship, indicating that the input of a metric calculation depends on the calculation result of another metric.

[0109] Acyclic: DAG does not allow loops, which ensures the sequential nature of computation and avoids infinite loops in computational tasks.

[0110] In one embodiment, see Figure 3 , the above-mentioned step S130 may include steps S131 to S134.

[0111] S131 , parsing the processing request to extract request rules and related user information.

[0112] In this embodiment, a processing request is received from an API or a message queue. Each request contains the following information:

[0113] User unique ID: used to identify the user who initiated the request.

[0114] Request rules: Contains the data processing rules requested by the user (for example, which indicators to select, how to combine indicators, which filtering conditions to use, etc.).

[0115] Topic Name: Specifies the subject or business scenario of the request.

[0116] This information will be parsed and used in subsequent data processing.

[0117] S132: Search for an indicator group that meets the requirements from the indicator configuration and rule group information according to the request rule.

[0118] In this embodiment, based on the request rules extracted from the request, the master control service searches the indicator configuration center for indicator groups that meet the requirements. These indicator groups have been pre-defined in the configuration database and are usually grouped according to business needs and rules. During this process:

[0119] The master control service will match the rule set in the indicator group with the rules requested by the user and filter out the indicator groups that meet the conditions.

[0120] If a matching indicator group is found, processing continues; otherwise, an error or unprocessable status is returned.

[0121] S133. Extract and encapsulate corresponding metadata for each indicator in the indicator group to obtain a structured data packet.

[0122] In this embodiment, each indicator in the selected indicator group is processed:

[0123] For each indicator, extract its related metadata information (such as indicator unique identifier, calculation expression, filter conditions, data source, parent indicator, etc.).

[0124] These metadata are encapsulated into structured data packets for subsequent indicator calculation and processing.

[0125] This structured data package will contain all the necessary information to ensure that the indicator configuration is correctly applied in subsequent calculation tasks.

[0126] Specifically, the relevant request rules are first extracted from the user request. These rules may include business conditions, time ranges, data types, and more. By parsing the parameters in the user request, the master control service extracts a rule set. For example, request rules may include user ID, time period, business type, data volume threshold, and more.

[0127] The master control service obtains predefined indicator groups from the indicator configuration center (usually a configuration database or indicator management system). The indicator group contains multiple indicators and information such as calculation rules, filter conditions, and data sources for each indicator.

[0128] Each metric group definition contains a set of rules, which are typically grouped based on business requirements and data analysis needs. The rule set within a metric group can include multiple conditions, which are used when matching.

[0129] The master service matches the rules extracted from the request with the rule set in the indicator group. The matching logic usually includes the following aspects:

[0130] Business condition matching: For example, the business type in the user request must be consistent with the business type of the indicator group.

[0131] Time range matching: For example, the time period specified in the request (such as a date range or hour range) must match the time conditions defined by the metric group.

[0132] Data volume threshold matching: For example, a request may have a specified data volume range, and the indicator group will define different data volume ranges for filtering based on business needs.

[0133] Other matching conditions: including user ID, region, device type, and other business-related parameters.

[0134] Specific matching method:

[0135] A metric group is considered to match a request rule if both the request rule and the rules in the metric group meet all conditions.

[0136] If there are multiple rule conditions, there may be "and" or "or" matching logic. For example:

[0137] And Match: All conditions must be met at the same time.

[0138] Or match: as long as any one of the conditions is met.

[0139] The master service compares each rule set in the indicator group one by one to select the indicator groups that meet the conditions. If there are multiple indicator groups that meet the conditions, the master service can further filter or return multiple matching results based on priority or other rules.

[0140] The screening process may involve matching calculation expressions, screening conditions, data sources, and other details of multiple indicator groups. If the request rules differ in some details, the system may return an unmatched result or an error message.

[0141] If a matching indicator group is found, the master control service will continue processing and enter the subsequent indicator calculation and processing process.

[0142] If no matching indicator group is found, the master service will return an error status or an "unable to process" prompt based on the system design. For example:

[0143] Returns an error: such as "No indicator group matching the requested criteria was found."

[0144] Returns an unprocessable status: such as "No suitable indicator available".

[0145] To improve matching efficiency, you can preprocess the rule set, such as creating an index and caching common rules. For complex business rules, consider introducing a fuzzy matching strategy to more flexibly handle user requests. For multiple matching indicator groups, you can introduce priority rules to select the optimal indicator group for subsequent processing.

[0146] For example, suppose a user request contains the following rules:

[0147] User ID = 12345;

[0148] Time period = 2025-03-01 to 2025-03-10;

[0149] Business type = A;

[0150] Then, the system queries the indicator configuration center, assuming there are the following two indicator groups:

[0151] Indicator group 1: User ID range: 10000-20000; Time period: 2025-03-01 to 2025-03-10; Business type: A;

[0152] Indicator Group 2:

[0153] User ID range: 20000-30000; Time period: 2025-03-05 to 2025-03-15; Business type: B.

[0154] In this case, metric group 1 will match the request, but metric group 2 will not (because the business type is different).

[0155] Through this rule matching approach, the master control service can efficiently filter out indicator groups that meet user requests from the indicator configuration center and provide support for subsequent indicator calculation and data processing.

[0156] S134. Analyze the dependency relationship of each indicator in the indicator group, construct a dependency graph between the indicators, traverse each indicator in the indicator group, add the indicator as a node, and establish dependency edges to obtain a directed acyclic graph of the dependency relationship.

[0157] In this embodiment, the dependency relationship of each indicator in the indicator group is analyzed, and the dependency graph between the indicators is constructed using the DAG package in Python. Each indicator in the indicator group is traversed, the indicator is added as a node using the add_node() method, and the dependency edge is established through the add_edge() method to obtain a directed acyclic graph of the dependency relationship.

[0158] In this example, a directed acyclic graph (DAG) is constructed by analyzing the dependencies between each indicator in the indicator group. This graph records the dependencies between all indicators and ensures the correctness of the calculation order:

[0159] Each metric is added to the DAG as a node.

[0160] According to the parent-child relationship, a directed edge is established between indicators through the add_edge() method, indicating that a certain indicator depends on the calculation result of another indicator.

[0161] In terms of implementation, the system uses the DAG package in Python (such as networkx or similar libraries) to build and operate DAG. The DAG package provides add_node() and add_edge() methods to add nodes and edges, ensuring that dependencies are correctly constructed.

[0162] Example dependencies:

[0163] Assume there is a metric group containing the following metrics:

[0164] Indicator A: Calculate the sum of a certain data.

[0165] Indicator B: Depends on indicator A and performs an average calculation based on it.

[0166] In a DAG, metrics A and B are two nodes connected by a directed edge, indicating that metric B depends on the result of metric A.

[0167] For each indicator, check if it has a parent indicator. If so, use the add_edge() method to establish a dependency relationship between the parent indicator and the current indicator in the DAG.

[0168] For each indicator, use the add_node() method to add it as a node in the DAG.

[0169] Iterate over all metrics, ensuring all dependencies are represented correctly.

[0170] In this embodiment, the above-mentioned step S130 reads the message from the message queue (MQ) and parses the user's request rules and related user information. Based on the data in the request rules, the indicator group stored in the memory is matched, the corresponding indicator group is found, and all related indicators in the group are associated with the indicator group ID. Next, a unique identifier is generated using the UUID and bound to the currently requested indicator group. This unique identifier will be used later when processing is completed to determine whether the result belongs to the indicator group, thereby confirming whether all indicators in the indicator group have been processed.

[0171] Parse the message body to obtain the user's unique identifier. Based on this identifier, execute a SQL query to count the user's data volume in a specific business table. Set an appropriate data volume threshold based on business requirements and determine which threshold range the user's data volume falls into. Next, classify and label all user metrics based on the data volume threshold and distribute them to different compute nodes for processing based on this classification.

[0172] Each metric contains the following metadata: the unique identifier of the metric group, the unique identifier of the metric (a different ID is generated each time), the metric's calculation expression, filter conditions, data source, parent metric, metric name, timeout period, data volume identifier, and processing status. This metadata is encapsulated into a structured data package that describes the metric's properties, dependencies, and execution context, supporting metric calculation, scheduling, and status management.

[0173] To manage the dependencies between indicators, the Directed Acyclic Graph (DAG) package in Python is used to record these dependencies. By traversing all indicators within an indicator group, each indicator is added as a node in the graph using the add_node() method of the DAG object. Dependencies between nodes are established using the add_edge() method, thus constructing a complete directed acyclic graph. If a certain indicator fails during subsequent processing, the system uses the get_predecessors() method to identify all of its predecessor nodes (i.e., all child indicators of its parent indicator) and mark the processing status of these predecessors as failed, thereby terminating invalid calculations and ensuring efficient resource utilization and process accuracy.

[0174] In this way, the system can efficiently manage and schedule the calculation tasks of indicators, ensuring that each indicator is correctly calculated and returned according to the dependency relationship.

[0175] S140: Utilize the directed acyclic graph to obtain indicators to be processed, and generate indicator processing and distribution tasks.

[0176] In this embodiment, the indicator to be processed refers to an indicator that needs to be processed.

[0177] Indicator processing and distribution tasks refer to tasks distributed to computing nodes.

[0178] In one embodiment, see Figure 4 , the above-mentioned step S140 may include steps S141 to S142.

[0179] S141. Obtain all leaf nodes that do not depend on other indicators from the directed acyclic graph through the all_leaves() method of DAG to obtain the indicator to be processed.

[0180] In this embodiment, all indicators to be processed can be obtained by calling the DAG.all_leaves() method. The specific steps are as follows:

[0181] Traverse the entire DAG and identify all leaf nodes.

[0182] These leaf nodes are the indicators to be processed. Since they do not have any dependencies, the system can start processing them immediately.

[0183] In data processing systems, the DAG structure is often used to express the dependencies between various computing tasks. Through the DAG, it is easy to find which tasks have no dependencies (i.e., leaf nodes), which can usually be processed independently.

[0184] In a DAG, leaf nodes are nodes that have no dependencies on them. In indicator processing tasks, leaf nodes are indicators to be processed. These indicators can be processed independently because they have no dependencies.

[0185] This method traverses the DAG structure and returns all indicator nodes (i.e. leaf nodes) that do not have any subsequent dependencies. These nodes represent indicators that can be processed, usually those that have no previous calculation dependencies.

[0186] During data processing, tasks without any pre-depen- dents (leaf nodes) are typically processed first, while other dependent tasks must wait for the completion of their predecessors before continuing. This strategy ensures that the order and dependencies of data processing are strictly adhered to.

[0187] S142: Pack the indicators to be processed and generate indicator processing and distribution tasks.

[0188] In step S141, all indicators to be processed are obtained through the DAG.all_leaves() method. These indicators no longer have any pre-dependencies. The next step is to package these indicators to be processed in order to generate a processing and distribution task.

[0189] The main purpose of the packaging operation is to combine all the indicators to be processed according to certain rules to form a complete task package. This task package includes the following:

[0190] Indicator Identifier: Each indicator has a unique identifier to distinguish different indicators.

[0191] Calculation Expression: The calculation logic or expression corresponding to each indicator. These expressions can be used to perform actual calculations and processing during data processing.

[0192] Data source information: Each indicator needs to obtain data from different data sources, and this information will be included in the task package.

[0193] Filter conditions: Indicators may have specific filter conditions for preprocessing data, which will also be included in the task package.

[0194] The packaged task set is usually a list of multiple indicators that can be processed in parallel.

[0195] Once the metrics to be processed are packaged, the next step is to generate an "indicator processing and distribution task." This step distributes the packaged metrics so that they can be processed by the computing nodes in the system.

[0196] Task generation: The generated indicator processing and distribution task will contain the indicator information to be processed, as well as all necessary information related to each indicator (such as data source, calculation expression, filtering conditions, etc.).

[0197] Distribution strategy: Based on the type and number of compute nodes, the system will distribute tasks to appropriate compute nodes according to a certain strategy. For example, the most suitable node may be selected based on factors such as the compute node load, type (CPU compute node, GPU compute node, etc.), and storage capacity.

[0198] Tasks can be distributed in different ways, either by broadcasting to all computing nodes or by dynamically assigning them to different nodes based on a load balancing strategy.

[0199] This distribution method supports rapid horizontal expansion. For example, if computing demand increases, new nodes can be dynamically added to process tasks, and the system will automatically distribute new tasks to these nodes.

[0200] After completing the indicator processing, each computing node will feedback the results to the main control system (Master node). This feedback usually includes information about the success or failure of the processing, and may also include error information.

[0201] In this embodiment, the goal of step S140 is to obtain the metrics to be processed through the DAG and generate and distribute the related processing tasks. The all_leaves() method identifies all leaf nodes without dependencies, i.e., the metrics to be processed. These metrics are then packaged into tasks, distributed, and ultimately processed by different compute nodes. This process not only ensures the correct handling of dependencies but also supports rapid horizontal expansion of compute nodes, improving the system's processing power and flexibility.

[0202] S150: Send the indicator processing and distribution task to the corresponding computing node, so that the computing node performs indicator processing and feeds back the processing result.

[0203] In this embodiment, the processing result refers to the result obtained by the computing node processing the task.

[0204] Specifically, the indicator processing and distribution task is sent to the corresponding computing node, so that the computing node puts the indicator processing and distribution task into the queue to be processed, and uses the Pandas library and predefined calculation methods to process the tasks in the queue to be processed. During the processing, the computing node calculates the task according to the expression and screening conditions of the indicator. If the processing is successful, the status of the indicator is updated to success, and the processing result is fed back; if the processing fails, the status of the indicator is updated to failure, and the processing result is fed back; wherein, when the computing node completes the processing of the indicator, it will encapsulate the processing result into a data packet and return it through the UDP protocol.

[0205] After receiving the indicator processing task from the upstream data source or external request, the computing node will distribute the task to the corresponding computing node based on the content of the task.

[0206] Each compute node has an independent thread that listens for server requests, which are typically received via a network protocol (such as UDP). These requests contain the indicator data packets to be processed, including the indicator's unique identifier, data source identifier, expression, filter conditions, and so on.

[0207] Once the computing node receives the indicator data packet, it will add the task to the processing queue, and the internal consumer thread will take it out of the queue one by one for processing.

[0208] To ensure the efficiency and sequentiality of data processing, computing nodes usually use thread pools or asynchronous task queues to handle these tasks.

[0209] Each task is processed using predefined calculation methods and expressions (such as SQL-like expressions, filter conditions, etc.).

[0210] Use the Pandas library to operate on data in memory. Pandas provides efficient data filtering, data cleaning, and calculation functions. For example, compute nodes can use Pandas' eval function to convert expression strings into executable functions, and then process the data using Pandas' DataFrame.

[0211] If the task does not encounter any errors during the processing, the computing node will update the processing status to "success" and attach the result data (such as the calculated indicator value or data transformation result).

[0212] Failure: If an error occurs during the task processing (such as a calculation error, data source reading error, etc.), the computing node will update the processing status to "failed" and record the error information for subsequent analysis.

[0213] After the task processing is completed, the computing node will encapsulate the processing results into a data packet. The data packet will include key information such as:

[0214] Unique identifier of the indicator group;

[0215] Unique indicator identifier;

[0216] Indicator processing results;

[0217] Processing status.

[0218] The computing node then sends the result data packet back to the server via the UDP protocol.

[0219] S160, receiving the processing result, and caching the processing result in a temporary storage, having a third-party service persist the processing result to ElasticSearch, notifying the user of the corresponding processing result through a message queue, and returning the processing result when the user queries.

[0220] In this embodiment, the processing results are received, the status of the indicators are processed according to the dependency relationship of the directed acyclic graph, and the processing results are cached in a temporary storage device. The processing results are persisted to ElasticSearch by a third-party service, and the user is notified of the corresponding processing results through a message queue, and the processing results are returned when the user queries.

[0221] Specifically, after receiving the processing results returned by the computing node, the server first performs preliminary processing on the results.

[0222] When receiving results, the server uses a directed acyclic graph (DAG) approach to check the processing status based on the dependencies between indicators. If the processing of an indicator fails, the server uses the get_predecessors() method to iterate over its predecessors and mark the status of the relevant predecessors as failed. This prevents error propagation and stops further processing.

[0223] All failed tasks will be placed in a failure queue for recording and subsequent processing. Successful tasks will be stored in a success queue.

[0224] The server determines whether processing for a metric group is complete by identifying the unique identifier for that metric group in the result packet. Specifically, the server records the number of tasks for that metric group and compares it with the number of successful and failed tasks it has received. If not, the server waits until all metrics are processed.

[0225] When the processing status of all indicators (whether successful or failed) is updated, the master will perform the final write operation. All results (whether successful or failed) will eventually be saved to temporary storage (such as Redis).

[0226] Once all processing results are cached in temporary storage (such as Redis), the Master node will notify the third-party service (usually the Work service) to persist these results.

[0227] The Work service monitors data changes in Redis and writes new results to ElasticSearch when they are found. As a powerful search and analysis engine, ElasticSearch can efficiently store and query these results.

[0228] After the persistence process is completed, the Work service notifies the user of the task processing results through a message queue (such as Kafka, RabbitMQ, etc.).

[0229] These notifications will include the status of the processing results (success or failure) and possible error information. Through the message queue, users can receive feedback on the processing results in a timely manner.

[0230] After receiving the notification of the processing results, the user service will query the final processing results in ElasticSearch based on the information in the notification.

[0231] Users can use the query interface to obtain specific indicator processing results, whether they are successful results or failed log information.

[0232] This system design can efficiently handle a large number of indicator tasks and ensure task dependencies and status synchronization between various computing nodes. At the same time, it uses Redis and ElasticSearch to ensure data persistence and efficient query, meeting the needs of large-scale data processing and real-time monitoring.

[0233] The above-mentioned third-party service refers to WORD service.

[0234] Metrics processing and data processing are implemented by configuring a management center, a master control service, compute nodes, a data writing service, intermediate storage, and a message queue. The master control service distributes tasks to the compute nodes via the UDP protocol. After completing the tasks, the compute nodes return the results to the master control service for aggregation. The system's intermediate storage uses Redis to temporarily store data, ensuring that data is not lost during processing. Furthermore, the system features horizontal scalability, allowing computing nodes to be added to meet growing business needs, ensuring that the system's processing capacity can scale with increasing load.

[0235] This approach encompasses configuration management, task distribution and computation, result aggregation and storage, as well as dynamic system scheduling and horizontal scalability. Tasks are flexibly configured through a configuration management center, while a master control service distributes tasks and aggregates and stores computation results. Furthermore, the system supports dynamic scheduling, adjusting load based on business needs, and offers horizontal scalability, allowing the system's processing power to be increased by adding compute nodes.

[0236] The above-mentioned indicator processing scheduling method based on distributed computing realizes flexible indicator configuration and rule matching by receiving computing node registration requests and regularly synchronizing configuration and rule group information; it parses the user's processing requests and constructs a directed acyclic graph to optimize the dependency of indicators, thereby generating efficient indicator processing tasks and distributing the tasks to computing nodes for execution; the processing results are temporarily cached and persisted to ElasticSearch by a third-party service, and the user is notified and the results are returned in a timely manner through the message queue; the design of this process not only optimizes the indicator processing process, improves computing efficiency and real-time performance, but also supports dynamic selective computing and resource optimization allocation, thereby reducing maintenance costs, and can flexibly respond to complex business needs and efficient processing in big data scenarios.

[0237] Figure 5 FIG is a schematic block diagram of an indicator processing scheduling system 300 based on distributed computing provided by an embodiment of the present invention. Figure 5As shown, corresponding to the above index processing scheduling method based on distributed computing, the present invention also provides an index processing scheduling system 300 based on distributed computing. The index processing scheduling system 300 based on distributed computing includes a unit for executing the above index processing scheduling method based on distributed computing, and the system can be configured in a server. Specifically, please refer to Figure 5 The index processing scheduling system 300 based on distributed computing includes a periodic updating unit 301 , a request obtaining unit 302 , a parsing unit 303 , an obtaining unit 304 , a sending unit 305 and a receiving unit 306 .

[0238] The periodic update unit 301 is used to receive registration requests from computing nodes and regularly synchronize indicator configurations and rule group information from the synchronization management center; the request acquisition unit 304302 is used to obtain processing requests sent by users; the parsing unit 303 is used to parse the processing requests, match indicator groups according to the indicator configuration and rule group information, encapsulate the indicator information corresponding to the indicator group, and construct a directed acyclic graph of dependencies; the acquisition unit 304 is used to use the directed acyclic graph to obtain indicators to be processed and generate indicator processing distribution tasks; the sending unit 305 is used to send the indicator processing distribution tasks to the corresponding computing nodes, so that the computing nodes perform indicator processing and feedback the processing results; the receiving unit 306 is used to receive the processing results and cache the processing results in a temporary storage device. The third-party service persists the processing results to ElasticSearch, notifies the user of the corresponding processing results through the message queue, and returns the processing results when the user queries.

[0239] In one embodiment, the parsing unit 303 includes:

[0240] An extraction subunit is used to parse the processing request to extract the request rules and related user information; a search subunit is used to search for an indicator group that meets the requirements from the indicator configuration and rule group information according to the request rules; an encapsulation subunit is used to extract and encapsulate the corresponding metadata for each indicator in the indicator group to obtain a structured data packet; a graph construction subunit is used to parse the dependency relationship of each indicator in the indicator group, construct a dependency graph between the indicators, traverse each indicator in the indicator group, add the indicator as a node, and establish dependency edges to obtain a directed acyclic graph of the dependency relationship.

[0241] In one embodiment, the graph construction subunit is used to parse the dependency relationship of each indicator in the indicator group, use the DAG package in Python to build a dependency graph between the indicators, traverse each indicator in the indicator group, use the add_node() method to add the indicator as a node, and establish a dependency edge through the add_edge() method to obtain a directed acyclic graph of the dependency relationship.

[0242] In one embodiment, the acquiring unit 304 includes:

[0243] The indicator acquisition subunit is used to obtain all leaf nodes that do not depend on other indicators from the directed acyclic graph through the all_leaves() method of DAG to obtain the indicators to be processed; the task generation subunit is used to package the indicators to be processed and generate indicator processing and distribution tasks.

[0244] In one embodiment, the sending unit 305 is used to send the indicator processing and distribution task to the corresponding computing node, so that the computing node puts the indicator processing and distribution task into the queue to be processed, and uses the Pandas library and predefined calculation methods to process the tasks in the queue to be processed. During the processing, the computing node calculates the task according to the expression and screening conditions of the indicator. If the processing is successful, the status of the indicator is updated to success, and the processing result is fed back; if the processing fails, the status of the indicator is updated to failure, and the processing result is fed back; wherein, when the computing node completes the processing of the indicator, it will encapsulate the processing result into a data packet and return it through the UDP protocol.

[0245] In one embodiment, the receiving unit 306 is used to receive the processing results, process the status of the indicators according to the dependency relationship of the directed acyclic graph, and cache the processing results in a temporary storage device. The third-party service persists the processing results to ElasticSearch, notifies the user of the corresponding processing results through a message queue, and returns the processing results when the user queries.

[0246] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned indicator processing scheduling system 300 based on distributed computing and each unit can refer to the corresponding description in the aforementioned method embodiment. For the convenience and conciseness of the description, it will not be repeated here.

[0247] The above-mentioned index processing scheduling system 300 based on distributed computing can be implemented in the form of a computer program. The computer program can be used in Figure 6 Runs on the computer equipment shown.

[0248] See also Figure 6 , Figure 61 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0249] See Figure 6 The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0250] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, can enable the processor 502 to execute an indicator processing scheduling method based on distributed computing.

[0251] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0252] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an indicator processing scheduling method based on distributed computing.

[0253] The network interface 505 is used to communicate with other devices through the network. Figure 6 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0254] The processor 502 is configured to execute a computer program 5032 stored in the memory to implement the following steps:

[0255] Receive registration requests from computing nodes, and regularly synchronize indicator configurations and rule group information from the synchronization management center; obtain processing requests sent by users; parse the processing requests, match indicator groups according to the indicator configuration and rule group information, encapsulate indicator information corresponding to the indicator groups, and build a directed acyclic graph of dependencies; use the directed acyclic graph to obtain indicators to be processed, and generate indicator processing and distribution tasks; send the indicator processing and distribution tasks to the corresponding computing nodes, so that the computing nodes perform indicator processing and feedback the processing results; receive the processing results, cache the processing results in a temporary storage device, and have a third-party service persist the processing results to ElasticSearch, notify the user of the corresponding processing results through the message queue, and return the processing results when the user queries.

[0256] The indicator configuration includes English name, Chinese name, data source, screening condition, expression, and parent indicator; the rule group information includes information corresponding to a set of multiple indicators.

[0257] In one embodiment, when the processor 502 implements the steps of parsing the processing request, matching the indicator group according to the indicator configuration and rule group information, encapsulating the indicator information corresponding to the indicator group, and constructing a directed acyclic graph of the dependency relationship, the processor 502 specifically implements the following steps:

[0258] Parse the processing request to extract request rules and related user information; search for an indicator group that meets the requirements from the indicator configuration and rule group information according to the request rules; extract and encapsulate corresponding metadata for each indicator in the indicator group to obtain a structured data packet; parse the dependency relationship of each indicator in the indicator group, build a dependency graph between the indicators, traverse each indicator in the indicator group, add the indicator as a node, and establish dependency edges to obtain a directed acyclic graph of the dependency relationship.

[0259] In one embodiment, when the processor 502 implements the steps of parsing the dependency relationship of each indicator in the indicator group, constructing a dependency graph between the indicators, traversing each indicator in the indicator group, adding the indicator as a node, and establishing dependency edges to obtain a directed acyclic graph of the dependency relationship, the processor 502 specifically implements the following steps:

[0260] Analyze the dependency relationship of each indicator in the indicator group, use the DAG package in Python to build a dependency graph between the indicators, traverse each indicator in the indicator group, use the add_node() method to add the indicator as a node, and use the add_edge() method to establish a dependency edge to obtain a directed acyclic graph of the dependency relationship.

[0261] In one embodiment, when the processor 502 implements the step of obtaining the indicators to be processed by using the directed acyclic graph and generating the indicator processing and distribution task, it specifically implements the following steps:

[0262] All leaf nodes that do not depend on other indicators are obtained from the directed acyclic graph through the all_leaves() method of DAG to obtain the indicators to be processed; the indicators to be processed are packaged to generate indicator processing and distribution tasks.

[0263] In one embodiment, when the processor 502 implements the step of sending the indicator processing distribution task to the corresponding computing node so that the computing node performs indicator processing and feeds back the processing result, the processor 502 specifically implements the following steps:

[0264] Send the indicator processing and distribution task to the corresponding computing node, so that the computing node puts the indicator processing and distribution task into the queue to be processed, and uses the Pandas library and predefined calculation methods to process the tasks in the queue to be processed. During the processing, the computing node calculates the task according to the expression and screening conditions of the indicator. If the processing is successful, the status of the indicator is updated to success, and the processing result is fed back; if the processing fails, the status of the indicator is updated to failure, and the processing result is fed back; wherein, when the computing node completes the processing of the indicator, it will encapsulate the processing result into a data packet and return it through the UDP protocol.

[0265] In one embodiment, the processor 502 implements the steps of receiving the processing result, caching the processing result in a temporary storage, having a third-party service persist the processing result to ElasticSearch, notifying the user of the corresponding processing result through a message queue, and returning the processing result when the user queries, specifically implementing the following steps:

[0266] Receive the processing results, process the status of the indicators according to the dependency relationship of the directed acyclic graph, cache the processing results in a temporary storage, and have a third-party service persist the processing results to ElasticSearch, notify the user of the corresponding processing results through a message queue, and return the processing results when the user queries.

[0267] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0268] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0269] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor performs the following steps:

[0270] Receive registration requests from computing nodes, and regularly synchronize indicator configurations and rule group information from the synchronization management center; obtain processing requests sent by users; parse the processing requests, match indicator groups according to the indicator configuration and rule group information, encapsulate indicator information corresponding to the indicator groups, and build a directed acyclic graph of dependencies; use the directed acyclic graph to obtain indicators to be processed, and generate indicator processing and distribution tasks; send the indicator processing and distribution tasks to the corresponding computing nodes, so that the computing nodes perform indicator processing and feedback the processing results; receive the processing results, cache the processing results in a temporary storage device, and have a third-party service persist the processing results to ElasticSearch, notify the user of the corresponding processing results through the message queue, and return the processing results when the user queries.

[0271] The indicator configuration includes English name, Chinese name, data source, screening condition, expression, and parent indicator; the rule group information includes information corresponding to a set of multiple indicators.

[0272] In one embodiment, when the processor executes the computer program to implement the steps of parsing the processing request, matching the indicator group according to the indicator configuration and rule group information, encapsulating the indicator information corresponding to the indicator group, and constructing a directed acyclic graph of the dependency relationship, the processor specifically implements the following steps:

[0273] Parse the processing request to extract request rules and related user information; search for an indicator group that meets the requirements from the indicator configuration and rule group information according to the request rules; extract and encapsulate corresponding metadata for each indicator in the indicator group to obtain a structured data packet; parse the dependency relationship of each indicator in the indicator group, build a dependency graph between the indicators, traverse each indicator in the indicator group, add the indicator as a node, and establish dependency edges to obtain a directed acyclic graph of the dependency relationship.

[0274] In one embodiment, when the processor executes the computer program to implement the steps of parsing the dependency relationship of each indicator in the indicator group, constructing a dependency graph between the indicators, traversing each indicator in the indicator group, adding the indicator as a node, and establishing dependency edges to obtain a directed acyclic graph of the dependency relationship, the processor specifically implements the following steps:

[0275] Analyze the dependency relationship of each indicator in the indicator group, use the DAG package in Python to build a dependency graph between the indicators, traverse each indicator in the indicator group, use the add_node() method to add the indicator as a node, and use the add_edge() method to establish a dependency edge to obtain a directed acyclic graph of the dependency relationship.

[0276] In one embodiment, when the processor executes the computer program to implement the steps of obtaining the indicators to be processed by using the directed acyclic graph and generating the indicator processing and distribution tasks, the processor specifically implements the following steps:

[0277] All leaf nodes that do not depend on other indicators are obtained from the directed acyclic graph through the all_leaves() method of DAG to obtain the indicators to be processed; the indicators to be processed are packaged to generate indicator processing and distribution tasks.

[0278] In one embodiment, when the processor executes the computer program to implement the step of sending the indicator processing and distribution task to the corresponding computing node so that the computing node performs indicator processing and feeds back the processing result, the processor specifically implements the following steps:

[0279] Send the indicator processing and distribution task to the corresponding computing node, so that the computing node puts the indicator processing and distribution task into the queue to be processed, and uses the Pandas library and predefined calculation methods to process the tasks in the queue to be processed. During the processing, the computing node calculates the task according to the expression and screening conditions of the indicator. If the processing is successful, the status of the indicator is updated to success, and the processing result is fed back; if the processing fails, the status of the indicator is updated to failure, and the processing result is fed back; wherein, when the computing node completes the processing of the indicator, it will encapsulate the processing result into a data packet and return it through the UDP protocol.

[0280] In one embodiment, the processor executes the computer program to implement the steps of receiving the processing result, caching the processing result in a temporary memory, having a third-party service persist the processing result to ElasticSearch, notifying a user of the corresponding processing result through a message queue, and returning the processing result when the user queries, specifically implementing the following steps:

[0281] Receive the processing results, process the status of the indicators according to the dependency relationship of the directed acyclic graph, cache the processing results in a temporary storage, and have a third-party service persist the processing results to ElasticSearch, notify the user of the corresponding processing results through a message queue, and return the processing results when the user queries.

[0282] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0283] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0284] In the several embodiments provided herein, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementations may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0285] The steps in the method of the embodiment of the present invention may be adjusted in order, combined, or deleted as needed. The units in the system of the embodiment of the present invention may be combined, divided, or deleted as needed. In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0286] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention.

[0287] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. The index processing scheduling method based on distributed computing is characterized by: include: Receive registration requests from computing nodes and regularly synchronize indicator configuration and rule group information from the synchronization management center; Get the processing request sent by the user; Parsing the processing request, matching the indicator group according to the indicator configuration and rule group information, encapsulating the indicator information corresponding to the indicator group, and constructing a directed acyclic graph of the dependency relationship; Utilizing the directed acyclic graph to obtain indicators to be processed, and generating indicator processing and distribution tasks; Sending the indicator processing distribution task to the corresponding computing node, so that the computing node performs indicator processing and feeds back the processing result; The processing results are received and cached in a temporary storage. A third-party service persists the processing results to ElasticSearch, notifies the user of the corresponding processing results through a message queue, and returns the processing results when the user queries.

2. The index processing scheduling method based on distributed computing according to claim 1 is characterized in that: The indicator configuration includes English name, Chinese name, data source, screening condition, expression, and parent indicator; the rule group information includes information corresponding to a set of multiple indicators.

3. The index processing scheduling method based on distributed computing according to claim 1 is characterized in that: The parsing of the processing request, matching the indicator group according to the indicator configuration and rule group information, encapsulating the indicator information corresponding to the indicator group, and constructing a directed acyclic graph of the dependency relationship includes: Parsing the processing request to extract request rules and related user information; Searching for an indicator group that meets the requirements from the indicator configuration and rule group information according to the request rule; Extracting and encapsulating corresponding metadata for each indicator in the indicator group to obtain a structured data packet; The dependency relationship of each indicator in the indicator group is parsed, a dependency graph between the indicators is constructed, each indicator in the indicator group is traversed, the indicator is added as a node, and dependency edges are established to obtain a directed acyclic graph of the dependency relationship.

4. The index processing scheduling method based on distributed computing according to claim 3 is characterized in that: The step of parsing the dependency relationship of each indicator in the indicator group, constructing a dependency graph between the indicators, traversing each indicator in the indicator group, adding the indicator as a node, and establishing dependency edges to obtain a directed acyclic graph of the dependency relationship includes: Analyze the dependency relationship of each indicator in the indicator group, use the DAG package in Python to build a dependency graph between the indicators, traverse each indicator in the indicator group, use the add_node() method to add the indicator as a node, and use the add_edge() method to establish a dependency edge to obtain a directed acyclic graph of the dependency relationship.

5. The index processing scheduling method based on distributed computing according to claim 4 is characterized in that: The method of using the directed acyclic graph to obtain the indicators to be processed and generating indicator processing and distribution tasks includes: Obtain all leaf nodes that do not depend on other indicators from the directed acyclic graph through the all_leaves() method of DAG to obtain the indicators to be processed; The indicators to be processed are packaged to generate indicator processing and distribution tasks.

6. The index processing scheduling method based on distributed computing according to claim 1 is characterized in that: The sending of the indicator processing and distribution task to the corresponding computing node so that the computing node performs indicator processing and feeds back the processing result includes: Send the indicator processing and distribution task to the corresponding computing node, so that the computing node puts the indicator processing and distribution task into the queue to be processed, and uses the Pandas library and predefined calculation methods to process the tasks in the queue to be processed. During the processing, the computing node calculates the task according to the expression and screening conditions of the indicator. If the processing is successful, the status of the indicator is updated to success, and the processing result is fed back; if the processing fails, the status of the indicator is updated to failure, and the processing result is fed back; wherein, when the computing node completes the processing of the indicator, it will encapsulate the processing result into a data packet and return it through the UDP protocol.

7. The index processing scheduling method based on distributed computing according to claim 1 is characterized in that: The receiving of the processing result and caching the processing result in a temporary storage, having a third-party service persist the processing result to ElasticSearch, notifying the user of the corresponding processing result through a message queue, and returning the processing result when the user queries, including: Receive the processing results, process the status of the indicators according to the dependency relationship of the directed acyclic graph, cache the processing results in a temporary storage, and have a third-party service persist the processing results to ElasticSearch, notify the user of the corresponding processing results through a message queue, and return the processing results when the user queries.

8. The index processing scheduling system based on distributed computing is characterized by: include: Periodic update unit, used to receive registration requests from computing nodes and regularly synchronize indicator configuration and rule group information from the synchronization management center; A request acquisition unit, used to acquire a processing request sent by a user; a parsing unit, configured to parse the processing request, match an indicator group according to the indicator configuration and rule group information, encapsulate indicator information corresponding to the indicator group, and construct a directed acyclic graph of dependency relationships; An acquisition unit, configured to acquire the indicators to be processed using the directed acyclic graph and generate indicator processing and distribution tasks; a sending unit, configured to send the indicator processing distribution task to the corresponding computing node, so that the computing node processes the indicator and feeds back the processing result; The receiving unit is used to receive the processing results and cache the processing results in a temporary storage. The third-party service persists the processing results to ElasticSearch, notifies the user of the corresponding processing results through the message queue, and returns the processing results when the user queries.

9. The index processing scheduling system based on distributed computing according to claim 8 is characterized in that: The parsing unit includes: an extraction subunit, configured to parse the processing request to extract request rules and related user information; A search subunit, configured to search for an indicator group that meets the requirements from the indicator configuration and rule group information according to the request rule; The encapsulation subunit is used to extract and encapsulate corresponding metadata for each indicator in the indicator group to obtain a structured data packet; The graph construction subunit is used to parse the dependency relationship of each indicator in the indicator group, construct a dependency graph between the indicators, traverse each indicator in the indicator group, add the indicator as a node, and establish dependency edges to obtain a directed acyclic graph of the dependency relationship.

10. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

Images