A multi-party scheduled service data processing method and system

Abstract

The application provides a multi-party scheduled business data processing method and system, relates to the technical field of data processing, and obtains a plurality of to-be-scheduled tasks by defining business scene information collection business data; performs scheduling analysis according to the business scene and a plurality of information sources, constructs a load balancing strategy; schedules and distributes the plurality of to-be-scheduled tasks according to a message queue, and obtains a task distribution result; performs business processing identification according to the task distribution result, determines a plurality of to-be-processed nodes; synchronizes the plurality of to-be-processed business data to a data processing cloud for processing monitoring, traces back to the plurality of to-be-processed nodes according to a monitoring result, and generates a node trace back result; and intelligently optimizes and processes the plurality of to-be-processed business data according to the node trace back result. The application solves the technical problem that the existing method is insufficient in flexibility and adaptability and is difficult to completely adapt to changing business requirements and data environments, and improves the speed, efficiency and accuracy of business data processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a multi-party scheduling business data processing method and system. Background Technology

[0002] In today's data-driven business environment, multi-party scheduling of business data processing has become crucial for improving operational efficiency, optimizing resource allocation, and enhancing decision-making quality. Multi-party scheduling involves collecting, processing, and analyzing data from multiple sources to support various business decisions and operations. This data processing approach is widely used in industries such as financial services, supply chain management, and e-commerce.

[0003] Existing multi-party scheduling methods for business data processing include rule-based scheduling, distributed data processing platforms, machine learning-driven data processing, real-time data processing systems, and the utilization of cloud infrastructure. Rule-based scheduling relies on predefined rules and, while performing well in stable environments, its flexibility and adaptability are limited when facing complex and ever-changing business environments. Distributed platforms such as Hadoop and Spark, along with machine learning algorithms, can process large-scale datasets in parallel, but require significant computing resources and expertise. Machine learning algorithms can dynamically adjust scheduling strategies based on historical and real-time data. Real-time data processing systems are suitable for scenarios requiring rapid response, but may encounter performance bottlenecks in high-concurrency scenarios, affecting the timeliness of data processing. Cloud services offer flexible resource management and scalability, but cost and management complexity hinder their large-scale application. Summary of the Invention

[0004] This application provides a multi-party scheduling business data processing method and system, which solves the technical problems that existing multi-party scheduling business data processing methods are insufficient in terms of flexibility and adaptability, and are difficult to fully adapt to constantly changing business needs and data environments, thereby achieving the technical effect of improving the speed, efficiency and accuracy of business data processing.

[0005] In view of the above problems, this application provides a multi-party scheduling business data processing method, the method comprising: traversing multiple information sources to define business scenario information; collecting business data based on the business scenario to obtain multiple tasks to be scheduled; performing scheduling analysis based on the business scenario and the multiple information sources to construct a load balancing strategy; scheduling and allocating the multiple tasks to be scheduled according to message queues based on the load balancing strategy to obtain task allocation results; identifying business processing based on the task allocation results to determine multiple nodes to be processed, the multiple nodes to be processed containing multiple pieces of business data to be processed; synchronizing the multiple pieces of business data to be processed to a data processing cloud for processing monitoring; backtracking the multiple nodes to be processed based on the monitoring results to generate node backtracking results; and synchronizing the node backtracking results to the data processing cloud for intelligent optimization processing of the multiple pieces of business data to be processed.

[0006] On the other hand, this application also provides a multi-party scheduling business data processing system, the system comprising: a task acquisition module, which is used to traverse multiple information sources to define business scenario information, collect business data based on the business scenario, and obtain multiple tasks to be scheduled; a strategy construction module, which is used to perform scheduling analysis based on the business scenario and the multiple information sources, and construct a load balancing strategy; a scheduling allocation module, which is used to schedule and allocate the multiple tasks to be scheduled according to message queues based on the load balancing strategy, and obtain task allocation results; a node determination module, which is used to identify business processing based on the task allocation results, determine multiple nodes to be processed, the multiple nodes to be processed containing multiple business data to be processed; a node backtracking module, which is used to synchronize the multiple business data to be processed to a data processing cloud for processing monitoring, and backtrack the multiple nodes to be processed based on the monitoring results to generate node backtracking results; and a data optimization module, which is used to perform intelligent optimization processing on the multiple business data to be processed based on the node backtracking results synchronized to the data processing cloud.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages: By traversing multiple information sources to define business scenario information, the data types to be collected can be more accurately determined. Based on these business scenarios, business data is collected to obtain multiple tasks to be scheduled. Scheduling analysis is performed based on the business scenarios and the multiple information sources to construct a load balancing strategy. Based on the load balancing strategy, the multiple tasks to be scheduled are allocated according to message queues to obtain task allocation results. Intelligent scheduling ensures that tasks are allocated reasonably according to priority and resource status, improving processing efficiency. Based on the task allocation results, business processing is identified, determining multiple pending processing nodes, each containing multiple pending business data. Identifying business processing requirements and determining processing nodes helps to process data in a targeted manner. The multiple pending business data are synchronized to the data processing cloud for processing monitoring. Based on the monitoring results, the multiple pending nodes are traced back to generate node traceback results, which can promptly identify and resolve potential problems, ensuring the quality of data processing. Based on the node traceback results, the multiple pending business data are intelligently optimized and processed in the data processing cloud.

[0008] In summary, this application, through intelligent scheduling analysis and automated task allocation, constructs a load balancing strategy based on real-time business needs and data changes, ensuring fair distribution of tasks across multiple nodes and avoiding resource idleness and overuse. This not only improves resource utilization but also reduces the risk of single-point overload, making data processing more efficient and stable. Through real-time data processing and monitoring, it responds promptly to data changes and intelligently optimizes based on the processing results using the powerful processing capabilities of the cloud to meet the growth of business needs and increasing data volume. This not only improves the speed and depth of data processing but also provides excellent scalability, making data processing more accurate and efficient, while also accelerating business response speed. Therefore, this solution can better adapt to dynamically changing business environments, improving the flexibility and adaptability of business data processing.

[0009] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0010] Figure 1 A flowchart illustrating a multi-party scheduling business data processing method provided in an embodiment of this application; Figure 2 A flowchart illustrating the construction of a load balancing strategy in a multi-party scheduling business data processing method provided in this application embodiment; Figure 3A flowchart illustrating the generation of node backtracking results in a multi-party scheduling business data processing method provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of a multi-party scheduling business data processing system provided in an embodiment of this application.

[0011] Figure labeling: Task acquisition module 10, strategy construction module 20, scheduling and allocation module 30, node determination module 40, node backtracking module 50, data optimization module 60. Detailed Implementation

[0012] This application provides a multi-party scheduling business data processing method and system, which solves the technical problems that existing multi-party scheduling business data processing methods are insufficient in terms of flexibility and adaptability, and are difficult to fully adapt to constantly changing business needs and data environments, thereby achieving the technical effect of improving the speed, efficiency and accuracy of business data processing.

[0013] Example 1, as Figure 1 As shown in the figure, this application embodiment provides a multi-party scheduling business data processing method, the method including: Step S1: Traverse multiple information sources to define business scenario information, collect business data based on the business scenario, and obtain multiple tasks to be scheduled.

[0014] Specifically, information sources refer to various sources from which data can be collected, including databases, file systems, API interfaces, social media platforms, etc. Business scenarios refer to the specific business environment or application cases where data analysis and processing will be applied. Scheduled tasks refer to data processing tasks defined according to the business scenario that need to be automatically executed by the system.

[0015] In step S1, several business scenarios need to be defined first. These scenarios are typically related to the company's specific business needs, such as market analysis, risk assessment, and customer behavior analysis. Data is collected as comprehensively and diversely as possible by searching and examining different data sources, including internal databases, external data services, and user input. Based on the predefined business scenarios, data relevant to the business needs is filtered and extracted. This data is then organized into specific tasks, such as data cleaning or analysis tasks, and placed in a task queue awaiting scheduling.

[0016] This step ensures that data collection and processing are based on accurate and comprehensive business understanding, providing fundamental scenario reference information for subsequent data processing and analysis.

[0017] Step S2: Based on the business scenario and the multiple information sources, perform scheduling analysis and construct a load balancing strategy.

[0018] Specifically, load balancing is a strategy to optimize the allocation of data processing tasks. It aims to ensure that the load on all data processing resources (such as servers and CPUs) is as even as possible, avoiding situations where some resources are overloaded while others are idle, thereby improving overall processing efficiency and resource utilization.

[0019] First, it's necessary to analyze the characteristics of the business scenario, such as data volume, processing complexity, and real-time requirements, to determine the priority and resource needs of data processing tasks. Then, analyze the currently available information sources and computing resources, including server CPU, memory, and network bandwidth, to assess their processing capacity and load. Based on this information, construct a load balancing strategy. This strategy defines how to allocate tasks to different resources to ensure efficient resource utilization and rapid task processing. This process can use load balancing algorithms for task allocation, such as round-robin, weighted round-robin, and least-connections algorithms. Round-robin algorithms simply distribute tasks evenly across all resources; weighted round-robin algorithms allocate tasks based on resource processing capacity; and least-connections algorithms allocate tasks to the resource with the fewest connections currently available.

[0020] This step establishes a load balancing strategy to balance task loads while maximizing resource utilization, ensuring high-quality data processing services are provided across various business scenarios.

[0021] Step S3: Based on the load balancing strategy, schedule and allocate the multiple tasks to be scheduled according to the message queue to obtain the task allocation result.

[0022] Specifically, based on the load balancing strategy constructed in step S2, the tasks to be scheduled are analyzed to assess the resource requirements, priority, and relevance to the business scenario of each task. These tasks are then placed into a message queue in a specific order. A message queue is a data structure used to store and manage tasks awaiting processing. In a message queue, tasks are arranged in a certain order, typically following a first-in, first-out (FIFO) principle.

[0023] Based on the analysis results, tasks are retrieved from the message queue using the message queue's API or management tools. These tasks are then allocated to different resources according to a load balancing strategy to ensure fair and efficient distribution across processing nodes, ultimately yielding the task allocation result. This result, showing the resources allocated to each task, is the outcome of the scheduling process.

[0024] This step ensures efficient task allocation and execution, improves the throughput and response speed of the task processing system, and avoids resource overload.

[0025] Step S4: Based on the task allocation result, identify the business processing and determine multiple nodes to be processed, wherein the multiple nodes to be processed contain multiple business data to be processed.

[0026] Specifically, a business processing identifier is a unique marker used to identify a specific business processing task within the data processing flow. It contains information such as task type, priority, and data source, guiding the execution of data processing. A pending processing node refers to a computing resource or processing unit within the data processing flow that awaits the execution of a specific business data processing task. These nodes can be servers, virtual machines, containers, etc., and are responsible for executing the business data processing tasks specified by the task allocation results. Pending business data refers to the specific dataset waiting to be processed within the data processing flow. This data is divided and allocated to different processing nodes based on business scenarios and task requirements.

[0027] In step S4, the task allocation results are first received, and the tasks are then identified based on these results, including the task's business processing identifier, data source, and processing priority. Next, based on this information, it is determined which pending processing nodes will be responsible for executing which pending business data, and the relevant business data is synchronized to these pending processing nodes for subsequent processing. By determining the business processing identifier and the pending processing nodes, it is ensured that each business data processing task is executed accurately and efficiently on the designated processing node, thereby improving the efficiency and reliability of the data processing workflow.

[0028] For example, in an e-commerce order processing scenario, order processing tasks are allocated based on a load balancing strategy. A unique business processing identifier is generated for each order processing task. These identifiers contain information such as the order ID and processing priority. The system uses this information to determine which processing nodes (such as order processing servers) will be responsible for processing which orders. For instance, the task "ORD202308301234" might be assigned to processing nodes H1 and H2, with each node responsible for processing different order data.

[0029] Step S5: Synchronize the multiple pending business data to the data processing cloud for processing monitoring, and backtrack the multiple pending nodes based on the monitoring results to generate node backtracking results.

[0030] Specifically, cloud-based data processing refers to remote servers or computing resources provided by cloud service providers for processing and storing data. These resources offer powerful computing capabilities and storage space to support large-scale data processing needs. Node backtracking results refer to the specific information and cause analysis obtained after the node backtracking process, regarding node status, processing anomalies, and their causes, which are used for subsequent optimization and problem-solving.

[0031] The pending business data is synchronized to the cloud for processing and analysis. Simultaneously, the data processing process is monitored and analyzed to ensure its correctness and efficiency. This step utilizes cloud service monitoring tools to monitor the execution status and resource usage of data processing tasks in real time. Monitoring includes task progress, CPU and memory usage, network latency, and error logs. If anomalies are detected during monitoring, such as processing delays, resource bottlenecks, or data processing errors, the relevant processing nodes are traced back. This backtracking process involves examining node log files, performance data, and network connectivity to pinpoint the cause of the problem. For example, log analysis tools can be used for in-depth analysis of node logs to help identify the specific location and cause of processing anomalies. Finally, node backtracking results are generated to record and analyze problems encountered during data processing, ensuring transparency and traceability, and improving the efficiency and accuracy of data processing.

[0032] Step S6: Based on the node backtracking results, synchronize the data to the data processing cloud to perform intelligent optimization processing on the multiple pending business data.

[0033] Specifically, in step S6, based on the node backtracking results, performance data and error logs during the data processing process are analyzed to identify problems and potential areas for improvement. The cloud-based data processing system utilizes optimization algorithms and machine learning to optimize and improve the data processing process, formulating specific optimization strategies such as adjusting data partitioning and optimizing computing resource allocation. Optimization algorithms include, but are not limited to, data preprocessing, model parameter adjustment, and algorithm optimization. These optimization strategies are then applied to the data processing workflow for implementation, and data processing performance is continuously monitored to evaluate the optimization effects and ensure that the improvement measures achieve the expected goals, thereby improving the speed, efficiency, and accuracy of data processing.

[0034] For example, historical data can be used to train machine learning models to predict and identify optimization opportunities. In e-commerce recommendation systems, backtracking analysis reveals that some recommendation models are inefficient on specific datasets. Machine learning algorithms are used to analyze model performance, identifying redundant computations in the data preprocessing step. Optimizing the data preprocessing process reduces computation time and improves the overall response speed of the recommendation system.

[0035] Furthermore, such as Figure 2 As shown, step S2 in this embodiment further includes: The multiple information sources are evaluated based on the business scenario to generate multiple evaluation results. Weights are assigned to the multiple information sources according to the evaluation results to obtain multiple weight coefficients. The multiple information sources are then sorted in descending order according to the multiple weight coefficients to determine an information source sequence. The information source sequence is traversed to extract the first information source in the first position. Based on the first information source, resource load is evaluated according to the business scenario to determine the first resource load data. This process is iterated to complete the traversal of the information source sequence, obtaining the first resource load data, the second resource load data, ..., the Nth resource load data, where N is the total number of information source sequences and N is an integer greater than 1. A weighted round-robin approach is then performed based on the first resource load data, the second resource load data, ..., the Nth resource load data to construct the load balancing strategy.

[0036] Specifically, the first step is to evaluate the performance, data quality, and reliability of multiple information sources based on the business scenario, generating evaluation results. This can be done by monitoring and collecting performance metrics of the data sources using tools such as Prometheus or Grafana. The evaluation includes the reliability of the information sources, data quality, update frequency, and relevance to the business scenario. The evaluation results are used to determine the priority and weight of the information sources.

[0037] Based on the evaluation results, weight coefficients are assigned to the information sources. These weight coefficients reflect the importance of different information sources in the scheduling analysis. The setting of these weight coefficients must consider business needs and data source characteristics, and can be achieved through expert decision-making or automatic calculation by algorithms. After the weight coefficients are determined, multiple information sources are arranged in descending order of their weights, forming an information source sequence. This sequence determines the priority order of information sources during scheduling analysis, guiding the order of resource scheduling and optimization, and helping to ensure that the most important information sources are processed first when resources are limited.

[0038] The process iterates through the information source sequence, extracting the first information source and evaluating its resource load to determine the first resource load data. This process iterates through the next information source until all sources have been evaluated, resulting in a series of resource load data. This resource load data represents the resources consumed by each information source during data processing, such as CPU, memory, and bandwidth. It is a key indicator for evaluating information source performance and will be used to construct a load balancing strategy. Based on the obtained first, second, ... Nth resource load data, a load balancing strategy is constructed using a weighted round-robin method to ensure that the load balancing strategy can be dynamically adjusted according to the actual load of different information sources. Here, N is the total number of information source sequences, and N is an integer greater than 1.

[0039] By following the steps above, an efficient and fair load balancing strategy can be constructed based on the characteristics of the business scenario and information source, thereby optimizing the data processing flow and resource utilization.

[0040] Furthermore, step S3 in this embodiment includes: Based on the business scenario, a requirement analysis is performed to determine the business data requirement information; multiple initial queues are defined, and the throughput of the multiple initial queues is calculated based on the business data requirement information combined with the first resource load data, the second resource load data, ... the Nth resource load data; the multiple initial queues are integrated based on their throughput to determine the message queue.

[0041] Specifically, business data requirements are data needs determined based on requirements analysis, including the type, format, and processing method of the required data. A detailed requirements analysis is conducted based on the business scenario to determine the required types, formats, quantities, and processing speeds of the business data to be processed. Next, based on the business data requirements, multiple initial queues are defined, each representing a specific data processing flow. The throughput of each initial queue is calculated by combining the business data requirements with first resource load data, second resource load data, ..., Nth resource load data. Throughput refers to the system's or device's ability to process data per unit of time, typically measured in data transmitted per second. The throughput of multiple initial queues is integrated to form a unified processing flow, i.e., a message queue. Through these steps, an efficient and scalable message queue can be built based on the business scenario and data requirements, thereby optimizing the data processing flow and resource utilization.

[0042] For example, in an e-commerce platform, a requirements analysis is performed on business data such as order processing, payment verification, and inventory updates. The initial queue definition includes a queue for receiving orders, an order verification queue, and an inventory update queue. By calculating the throughput of these queues under different resource loads, a message queue system capable of efficiently processing orders is designed.

[0043] Furthermore, step S3 in this embodiment of the application also includes: The multiple tasks to be scheduled are dynamically evaluated according to the load balancing strategy to generate task load evaluation results; the task load evaluation results are matched with the throughput of the multiple initial queues to determine the task matching results; based on the task matching results, the multiple tasks to be scheduled are allocated to the message queue for task enqueueing to obtain the task allocation results.

[0044] Preferably, in step S3 of this embodiment, matching the task load assessment result with the throughput of the plurality of initial teams to determine the task matching result includes: Based on the task load assessment results, the multiple initial teams are calculated to determine multiple load ratios, which correspond to the multiple initial teams. The load files of historical scheduled tasks are retrieved, and the expected load data of the multiple tasks to be scheduled is determined based on the load files of the historical scheduled tasks. The multiple load ratios are compared with the expected load data, and the comparison results are matched with the throughput of the multiple initial teams to determine the task matching result.

[0045] Specifically, each task to be scheduled is first evaluated in real time according to the load balancing strategy. Considering the status of all available resources, such as CPU utilization, memory usage, and network latency, and combining this with the characteristics of the task to be scheduled, such as task type, estimated processing time, and required resources, a resource requirement overview for each task is generated. Based on this dynamic evaluation, the resource requirements and estimated processing time for each task are generated as the task load assessment result.

[0046] Based on the task load assessment results and the throughput of each initial queue, calculate the load ratio of each initial queue to each task. For example, the load ratio can be calculated as the ratio of task resource requirements to queue throughput. Retrieve load profiles of historically scheduled tasks, which detail past task execution, such as actual resource consumption and processing time. Use these historical scheduling profiles as training datasets to train machine learning models, such as linear regression or neural networks, to predict task resource requirements. Combined with the characteristics of the currently scheduled tasks, predict the expected resource consumption and processing time for each task, i.e., the expected load data.

[0047] The expected load data is compared with the load ratio to find combinations of tasks and queues with a load ratio close to 1 but not exceeding 1. If the load ratio is less than 1, it indicates that the queue throughput can meet the task requirements; if it is close to 1, it indicates a high degree of matching and high resource utilization efficiency. Based on the comparison results, the most suitable queue for each task is determined, forming a task matching result. According to the matching result, each task to be scheduled is assigned to the most suitable initial queue, completing the task enqueueing process.

[0048] The above steps can intelligently allocate tasks to the most suitable queue based on the task's load requirements and the initial queue's processing capacity, thereby achieving dynamic task allocation and optimizing data processing flow and resource utilization.

[0049] Furthermore, step S5 in this embodiment includes: The multiple pending business data are transmitted to the data processing cloud for data synchronization, generating multiple pending business data synchronization results. The data processing cloud then processes the multiple pending business data based on the synchronization results and the business data requirement information to construct a data processing module. The data processing module is monitored and recorded to obtain a monitoring log, which includes data processing monitoring time information, data processing throughput information, and data processing anomaly information. Based on the data processing monitoring time information and the data processing throughput information, the data processing anomaly information is identified, generating multiple anomaly tags. These anomaly tags are then added to the monitoring results.

[0050] Specifically, monitoring logs record information such as time, throughput, and anomalies during data processing for subsequent analysis and troubleshooting. Anomaly tags are used to identify abnormal events during data processing, helping to quickly locate and resolve data processing problems.

[0051] Multiple pending business data sets are transferred from local databases or data sources to a cloud-based data processing platform and synchronized, generating synchronization results to ensure data integrity and real-time performance. Next, a data processing module is built in the cloud based on business data requirements and the synchronized data. This module processes the data according to the synchronization results and the business data requirements. This data processing module is a collection of software components or algorithms used to process specific types of business data.

[0052] During data processing, the data processing module is monitored, and detailed information about the data processing is recorded, including the start and end times of data processing (monitoring time information), the amount of data processed (throughput information), and any abnormal events (abnormal information), generating a monitoring log. Based on the data processing monitoring time and throughput information, the log is analyzed, abnormal information is identified, and abnormal tags are generated. The generated abnormal tags are added to the monitoring results to facilitate subsequent analysis, reporting, and problem solving.

[0053] By following the steps above, data can be processed in the cloud environment based on business data requirements, and anomalies during the processing can be recorded and analyzed to optimize the data processing flow and improve data processing quality.

[0054] Furthermore, such as Figure 3 As shown, step S5 in this embodiment further includes: Based on the multiple anomaly tags, the multiple pending business data are matched and extracted to obtain multiple pending business anomaly data; backtracking conditions are set based on the multiple anomaly tags, and the backtracking conditions are triggered according to the multiple pending business anomaly data; backtracking analysis is performed by traversing the multiple pending nodes based on the backtracking conditions to determine multiple pending anomaly nodes; anomaly analysis is performed based on the multiple pending anomaly nodes to determine an anomaly analysis report; and the node backtracking result is constructed based on the multiple pending anomaly nodes and the anomaly analysis report.

[0055] Specifically, based on anomaly tags, datasets marked as anomalous are extracted from the pending business data, resulting in multiple pending business anomalous data sets. These anomalous data sets are business data tagged with anomaly tags that require special attention and processing. Backtracking conditions are set for each anomaly tag; these conditions are preset to trigger backtracking analysis, helping to pinpoint the nodes where the anomalies occurred. When pending business anomalous data with that anomaly tag triggers these conditions, backtracking analysis is initiated. Starting from the anomalous data that triggered the backtracking conditions, the nodes in the data processing flow are traversed backwards, analyzing the input and output of each node to determine the precise location of the anomaly. Through backtracking analysis, the source or key nodes of the anomalous data are identified, i.e., the pending anomalous nodes. In-depth analysis is performed on the identified anomalous nodes, including data status and processing logic, generating an anomaly analysis report. This report includes information such as the cause of the anomaly and its scope of impact. Combining the anomalous nodes and the analysis report forms a complete anomaly backtracking result, providing a basis for subsequent anomaly handling and optimization.

[0056] For example, an online advertising platform discovered an abnormal increase in click-through rates (CTR) while processing user click data. Using a "CTR Anomaly" tag, the platform extracted the abnormal dataset from a massive amount of click data. The backtracking condition was set to "CTR exceeding the average by 3 standard deviations." Starting from the click data marked as abnormal, the data flow was analyzed in reverse, examining each node in the data processing workflow, such as data reception, cleaning, and statistics. The analysis revealed an error in a filter rule setting at the data cleaning node, causing abnormal clicks to go unfiltered. An anomaly analysis report was generated, explaining the cause of the anomaly (incorrect filter setting) and its impact (affecting the accuracy of the CTR). Finally, the anomaly node (data cleaning node) and the anomaly analysis report (filter error leading to abnormal CTR) were integrated to form anomaly handling and prevention recommendations.

[0057] By following the steps above, we can not only quickly locate and resolve anomalies in data processing, but also optimize the data processing workflow based on anomaly analysis and improve data quality.

[0058] In summary, the multi-party scheduling business data processing method provided in this application has the following technical effects: By traversing multiple information sources to define business scenario information, the data types to be collected can be more accurately determined. Based on these business scenarios, business data is collected to obtain multiple tasks to be scheduled. Scheduling analysis is performed based on the business scenarios and the multiple information sources to construct a load balancing strategy. Based on the load balancing strategy, the multiple tasks to be scheduled are allocated according to message queues to obtain task allocation results. Intelligent scheduling ensures that tasks are allocated reasonably according to priority and resource status, improving processing efficiency. Based on the task allocation results, business processing is identified, determining multiple pending processing nodes, each containing multiple pending business data. Identifying business processing requirements and determining processing nodes helps to process data in a targeted manner. The multiple pending business data are synchronized to the data processing cloud for processing monitoring. Based on the monitoring results, the multiple pending nodes are traced back to generate node traceback results, which can promptly identify and resolve potential problems, ensuring the quality of data processing. Based on the node traceback results, the multiple pending business data are intelligently optimized and processed in the data processing cloud.

[0059] Overall, this application's embodiments, through intelligent scheduling analysis and automated task allocation, can construct load balancing strategies based on real-time business needs and data changes, ensuring fair distribution of tasks across multiple nodes and thus avoiding resource idleness and overuse. This not only improves resource utilization but also reduces the risk of single-point overload, making data processing more efficient and stable. Through real-time data processing and monitoring, it responds promptly to data changes and intelligently optimizes based on the processing results using the powerful processing capabilities of the cloud to meet the growth of business needs and the increase in data volume. This not only improves the speed and depth of data processing but also provides good scalability, making data processing more accurate and efficient, while also accelerating business response speed. Therefore, this solution can better adapt to dynamically changing business environments, improving the flexibility and adaptability of business data processing.

[0060] Example 2, as Figure 4 As shown in the figure, this application embodiment provides a multi-party scheduling business data processing system, the system comprising: The task acquisition module 10 is used to traverse multiple information sources to define business scenario information, collect business data based on the business scenario, and obtain multiple tasks to be scheduled.

[0061] The strategy construction module 20 is used to perform scheduling analysis based on the business scenario and the multiple information sources to construct a load balancing strategy.

[0062] The scheduling and allocation module 30 is used to schedule and allocate the multiple tasks to be scheduled according to the message queue based on the load balancing strategy, and obtain the task allocation result.

[0063] The node determination module 40 is used to identify business processing based on the task allocation result and determine multiple nodes to be processed, wherein the multiple nodes to be processed contain multiple business data to be processed.

[0064] The node backtracking module 50 is used to synchronize the multiple pending business data to the data processing cloud for processing and monitoring, and to backtrack the multiple pending nodes based on the monitoring results to generate node backtracking results.

[0065] The data optimization module 60 is used to intelligently optimize the multiple pending business data by synchronizing the node backtracking results to the data processing cloud.

[0066] Furthermore, the strategy construction module 20 in this embodiment is also used to perform the following steps: The multiple information sources are evaluated based on the business scenario to generate multiple evaluation results. Weights are assigned to the multiple information sources according to the evaluation results to obtain multiple weight coefficients. The multiple information sources are then sorted in descending order according to the multiple weight coefficients to determine an information source sequence. The information source sequence is traversed to extract the first information source in the first position. Based on the first information source, resource load is evaluated according to the business scenario to determine the first resource load data. This process is iterated to complete the traversal of the information source sequence, obtaining the first resource load data, the second resource load data, ..., the Nth resource load data, where N is the total number of information source sequences and N is an integer greater than 1. A weighted round-robin approach is then performed based on the first resource load data, the second resource load data, ..., the Nth resource load data to construct the load balancing strategy.

[0067] Furthermore, in this embodiment of the application, the scheduling and allocation module 30 is also used to perform the following steps: Based on the business scenario, a requirement analysis is performed to determine the business data requirement information; multiple initial queues are defined, and the throughput of the multiple initial queues is calculated based on the business data requirement information combined with the first resource load data, the second resource load data, ... the Nth resource load data; the multiple initial queues are integrated based on their throughput to determine the message queue.

[0068] Furthermore, in this embodiment of the application, the scheduling and allocation module 30 is also used to perform the following steps: The multiple tasks to be scheduled are dynamically evaluated according to the load balancing strategy to generate task load evaluation results; the task load evaluation results are matched with the throughput of the multiple initial queues to determine the task matching results; based on the task matching results, the multiple tasks to be scheduled are allocated to the message queue for task enqueueing to obtain the task allocation results.

[0069] Furthermore, in this embodiment of the application, the scheduling and allocation module 30 is also used to perform the following steps: Based on the task load assessment results, the multiple initial teams are calculated to determine multiple load ratios, which correspond to the multiple initial teams. The load files of historical scheduled tasks are retrieved, and the expected load data of the multiple tasks to be scheduled is determined based on the load files of the historical scheduled tasks. The multiple load ratios are compared with the expected load data, and the comparison results are matched with the throughput of the multiple initial teams to determine the task matching result.

[0070] Furthermore, in this embodiment, the node backtracking module 50 is also used to perform the following steps: The multiple pending business data are transmitted to the data processing cloud for data synchronization, generating multiple pending business data synchronization results. The data processing cloud then processes the multiple pending business data based on the synchronization results and the business data requirement information to construct a data processing module. The data processing module is monitored and recorded to obtain a monitoring log, which includes data processing monitoring time information, data processing throughput information, and data processing anomaly information. Based on the data processing monitoring time information and the data processing throughput information, the data processing anomaly information is identified, generating multiple anomaly tags. These anomaly tags are then added to the monitoring results.

[0071] Furthermore, in this embodiment, the node backtracking module 50 is also used to perform the following steps: Based on the multiple anomaly tags, the multiple pending business data are matched and extracted to obtain multiple pending business anomaly data; backtracking conditions are set based on the multiple anomaly tags, and the backtracking conditions are triggered according to the multiple pending business anomaly data; backtracking analysis is performed by traversing the multiple pending nodes based on the backtracking conditions to determine multiple pending anomaly nodes; anomaly analysis is performed based on the multiple pending anomaly nodes to determine an anomaly analysis report; and the node backtracking result is constructed based on the multiple pending anomaly nodes and the anomaly analysis report.

[0072] Through the foregoing detailed description of a multi-party scheduling business data processing method, those skilled in the art can clearly understand that the multi-party scheduling business data processing system in this embodiment corresponds to the system disclosed in Embodiment 2. As it corresponds to the method disclosed in Embodiment 1, it has corresponding functional modules and beneficial effects. For relevant details, please refer to the method section.

[0073] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A business data processing method for multi-party scheduling, characterized in that, The method includes: The system iterates through multiple information sources to define business scenario information, collects business data based on the business scenarios, and obtains multiple tasks to be scheduled. Based on the business scenario and the multiple information sources, a scheduling analysis is performed to construct a load balancing strategy. Based on the load balancing strategy, the multiple tasks to be scheduled are scheduled and allocated according to the message queue to obtain the task allocation result. Based on the task allocation result, business processing identifiers are determined, and multiple nodes to be processed are identified. The multiple nodes to be processed contain multiple business data to be processed. The multiple pending business data are synchronized to the data processing cloud for processing and monitoring. Based on the monitoring results, the multiple pending nodes are traced back to generate node traceback results. Based on the node backtracking results, the data processing cloud is synchronized to perform intelligent optimization processing on the multiple pending business data. Based on the aforementioned business scenario and the multiple information sources, a scheduling analysis is performed to construct a load balancing strategy. The method includes: The multiple information sources are evaluated based on the aforementioned business scenario, generating multiple evaluation results. Based on the multiple evaluation results, the multiple information sources are weighted to obtain multiple weight coefficients; The multiple information sources are arranged in descending order according to the multiple weighting coefficients to determine the information source sequence; The first information source in the first order is extracted by traversing the information source sequence. Based on the first information source, resource load is evaluated according to the business scenario to determine the first resource load data. This process is repeated to complete the traversal of the information source sequence and obtain the first resource load data, the second resource load data, ... the Nth resource load data, where N is the total number of information source sequences and N is an integer greater than 1. The load balancing strategy is constructed by performing a weighted round-robin algorithm based on the first resource load data, the second resource load data, ... the Nth resource load data; The message queue, the method includes: Based on the aforementioned business scenario, a requirements analysis is conducted to determine the business data requirements. Define multiple initial teams, and calculate the throughput of the multiple initial teams based on the business data demand information combined with the first resource load data, the second resource load data, ... the Nth resource load data; The message queue is determined by integrating the multiple initial queues based on their throughput.

2. The business data processing method for multi-party scheduling as described in claim 1, characterized in that, Based on the load balancing strategy, the multiple tasks to be scheduled are scheduled and allocated according to message queues to obtain task allocation results. The method includes: The multiple tasks to be scheduled are dynamically evaluated according to the load balancing strategy to generate task load evaluation results. The task matching result is determined by matching the task load assessment result with the throughput of the multiple initial teams; Based on the task matching results, the multiple tasks to be scheduled are assigned to the message queue for task enqueueing, and the task assignment results are obtained.

3. The business data processing method for multi-party scheduling as described in claim 2, characterized in that, The task matching result is determined by matching the task load assessment result with the throughput of the multiple initial teams, and the method includes: Based on the task load assessment results, multiple load ratios are calculated for the multiple initial teams, and these multiple load ratios correspond to the multiple initial teams. Retrieve the load profiles of historical scheduled tasks, and determine the expected load data of the multiple tasks to be scheduled based on the load profiles of the historical scheduled tasks. The task matching result is determined by comparing the multiple load ratios with the expected load data and matching the comparison results with the throughput of the multiple initial teams.

4. The business data processing method for multi-party scheduling as described in claim 1, characterized in that, The method for synchronizing the multiple pending business data to the data processing cloud for processing and monitoring includes: The multiple pending business data are transmitted to the data processing cloud for data synchronization, generating multiple pending business data synchronization results; The data processing cloud platform processes the multiple pending business data in conjunction with the business data requirement information based on the synchronization results of the multiple pending business data, thereby constructing a data processing module. The data processing module is monitored and recorded to obtain a monitoring log, which includes data processing monitoring time information, data processing throughput information, and data processing anomaly information. Based on the data processing monitoring time information and the data processing throughput information, the abnormal information of the data processing is identified, and multiple abnormal tags are generated; Add the multiple anomaly labels to the monitoring results.

5. The business data processing method for multi-party scheduling as described in claim 4, characterized in that, Based on the monitoring results, backtracking is performed on the multiple nodes to be processed to generate node backtracking results. The method includes: Based on the multiple anomaly tags, the multiple pending business data are matched and extracted to obtain multiple pending business anomaly data. Based on the multiple anomaly labels, backtracking conditions are set, and backtracking conditions are triggered according to the multiple pending business anomaly data. Based on the backtracking conditions, the multiple pending nodes are traversed for backtracking analysis to determine multiple pending anomaly nodes. An anomaly analysis is performed on the multiple unprocessed anomaly nodes to determine the anomaly analysis report; The node backtracking results are constructed based on the multiple unprocessed abnormal nodes and the abnormal analysis report.

6. A business data processing system for multi-party scheduling, characterized in that, The system is used to execute a multi-party scheduling business data processing method according to any one of claims 1-5, the system comprising: The task acquisition module is used to traverse multiple information sources to define business scenario information, collect business data based on the business scenario, and obtain multiple tasks to be scheduled. A strategy construction module is used to construct a load balancing strategy by performing scheduling analysis based on the business scenario and the multiple information sources. The scheduling and allocation module is used to schedule and allocate the multiple tasks to be scheduled according to the message queue based on the load balancing strategy, and obtain the task allocation result. A node determination module is used to identify business processing based on the task allocation result and determine multiple nodes to be processed, wherein the multiple nodes to be processed contain multiple business data to be processed. The node backtracking module is used to synchronize the multiple pending business data to the data processing cloud for processing monitoring, and backtrack the multiple pending nodes based on the monitoring results to generate node backtracking results; The data optimization module is used to intelligently optimize the multiple pending business data by synchronizing the node backtracking results to the data processing cloud.

Images