Data batch processing method and system based on distributed scheduling

Abstract

The invention provides a data batch processing method and system based on distributed scheduling, and the method comprises the steps: obtaining multi-source heterogeneous data, and carrying out the preprocessing of the multi-source heterogeneous data, and forming a standardized data set; based on a preset first threshold value, performing preliminary anomaly detection on the data set by utilizing a screening process, and marking potential abnormal data; for the residual data set excluding the potential abnormal data, calculating an abnormal score of each data point in the residual data set by adopting an abnormal detection algorithm; according to a preset second threshold value, screening out data points of which the abnormal scores exceed the second threshold value, and marking the data points as high abnormal data; generating abnormal data; splitting the data set excluding the abnormal data into a plurality of data blocks, and distributing the plurality of data blocks to corresponding processing nodes for parallel processing based on a distributed task scheduling strategy; and summarizing the processing results of the processing nodes. The data processing speed and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data batch processing method and system based on distributed scheduling. Background Art

[0002] In the financial industry, data batch processing is essential to ensure the accuracy of transactions, the security of user information, and the efficiency of overall business operations. As the amount of data processed by financial institutions continues to grow, how to effectively manage and process this data has become an important challenge facing the industry. Currently, data processing in the financial industry mainly adopts the following methods: Manual processing mode: In this mode, data is first collected and summarized through tools such as Excel. Then, the built-in functions of Excel are used to process, filter and transform the data to generate the required data reports. Although this method provides high data security and is suitable for manual review of small-scale or sensitive data, its process is time-consuming, labor-intensive, inefficient, and difficult to adapt to the needs of large-scale data processing.

[0003] Batch processing mode: As a traditional data processing method, batch processing mode divides data into batches of fixed size and processes them one by one. This method is relatively simple to implement and is suitable for scenarios with low real-time requirements. However, it has limitations in processing speed. Especially when facing massive data, batch processing mode may not provide a fast enough response time, thus affecting the timeliness of data analysis and decision support capabilities.

[0004] Stream processing mode: Different from the first two modes, the stream processing mode focuses on real-time data processing and can process continuous input data streams instantly. This method not only improves the data processing speed, but also provides analysis results in the first time, greatly enhancing the timeliness and accuracy of decision-making. However, the deployment and maintenance costs of the stream processing mode are high, requiring a professional technical team to design, develop and support, and its complexity also brings higher implementation difficulties.

[0005] In summary, as the financial industry's demand for data processing efficiency and real-time performance increases, choosing the right data processing method becomes particularly important. Each method has its own unique advantages and applicable scenarios. Financial institutions should reasonably choose the most appropriate solution based on their specific needs and technical conditions. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a data batch processing method and system based on distributed scheduling, which are used to solve the technical problems in the prior art.

[0007] In a first aspect, the invention provides the following technical solution: a data batch processing method based on distributed scheduling, the method comprising: Acquire multi-source heterogeneous data, and pre-process the multi-source heterogeneous data to form a standardized data set; Based on a predetermined first threshold, using a screening process to perform preliminary anomaly detection on the data set to mark potential abnormal data; For the remaining data set excluding the potential abnormal data, an anomaly detection algorithm is used to calculate the anomaly score of each data point in the remaining data set; According to a preset second threshold, data points whose anomaly scores exceed the second threshold are screened out and marked as highly abnormal data; Combining the potential abnormal data and the highly abnormal data to generate abnormal data; Splitting the data set excluding the abnormal data into multiple data blocks, and assigning the multiple data blocks to corresponding processing nodes for parallel processing based on a distributed task scheduling strategy; The processing results of each processing node are summarized and a data report is generated and fed back to the user end or business system.

[0008] Compared with the prior art, the beneficial effects of the present application are as follows: by combining the screening process with the anomaly detection algorithm, abnormal data can be detected more comprehensively, which can not only effectively improve the accuracy of data cleaning, but also significantly reduce the analysis deviation caused by data anomalies; by allocating multiple data blocks to corresponding processing nodes for parallel processing based on a distributed task scheduling strategy, not only the speed and efficiency of data processing are improved, but also the flexibility, reliability and scalability of the system are enhanced, which better meets the needs of large-scale data processing.

[0009] Furthermore, the multi-source heterogeneous data includes business system databases and log files.

[0010] Furthermore, the step of preprocessing the multi-source heterogeneous data to form a standardized data set includes: Completing the missing values ​​of the multi-source heterogeneous data by using mean filling or interpolation methods; Normalizing the multi-source heterogeneous data by using standardized scaling or minimum and maximum scaling to fill missing values; Obvious abnormal data in the multi-source heterogeneous data after normalization processing is filtered out to form a standardized data set.

[0011] Furthermore, the screening process includes: Arrange the data set in ascending order, and take the data point at the first position in the sorted data set as the first judgment value; taking a data point at a second position in the sorted data set as a second judgment value, and calculating a trend value based on the first judgment value and the second judgment value; An upper threshold and a lower threshold are calculated based on the first judgment value, the second judgment value and the trend value; Potential abnormal data in the data set is screened out based on the upper threshold and the lower threshold using a comparison formula.

[0012] Furthermore, the comparison formula includes:

[0013]

[0014] in, Represented as the first judgment value, Represented as the second judgment value, Represents potential abnormal data, Expressed as the coefficient.

[0015] Furthermore, the anomaly detection algorithm includes: Randomly sub-sampling the remaining data set to construct multiple isolation trees; Calculating the path length of each data point in the remaining data set in each of the isolation trees, and calculating the average path length; The calculation formula is used to calculate based on the average path length to obtain an anomaly score.

[0016] Furthermore, the calculation formula includes:

[0017] in, Represents data points The anomaly score, represents the weight coefficient of the path length term, Represents data points The average path length among all isolation trees, represents the dynamic normalization factor, represents the weight coefficient of the distribution density term, represents the slope adjustment factor, Represents data points The local density of Represents the average density of the data set;

[0018] represents the dynamic normalization factor, represents the mixing coefficient, represents the harmonic number, represents the number of data sets, Represents the standard deviation of the path lengths of all data points.

[0019] In a second aspect, the invention provides the following technical solution: a data batch processing system based on distributed scheduling, the system comprising: A processing module, used for acquiring multi-source heterogeneous data and preprocessing the multi-source heterogeneous data to form a standardized data set; A detection module, configured to perform preliminary anomaly detection on the data set based on a predetermined first threshold value and mark potential abnormal data by using a screening process; A calculation module, configured to calculate an abnormality score of each data point in the remaining data set by using an abnormality detection algorithm for the remaining data set excluding the potential abnormal data; A screening module, configured to screen out data points whose anomaly scores exceed a preset second threshold value according to the second threshold value, and mark them as highly abnormal data; A merging module, used for merging the potential abnormal data and the highly abnormal data to generate abnormal data; A parallel module, used for splitting the data set excluding the abnormal data into multiple data blocks, and allocating the multiple data blocks to corresponding processing nodes for parallel processing based on a distributed task scheduling strategy; The feedback module is used to summarize the processing results of each processing node and generate a data report to feed back to the user end or business system.

[0020] In a third aspect, the invention provides the following technical solution: a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data batch processing method based on distributed scheduling as described above is implemented.

[0021] In a fourth aspect, the invention provides the following technical solution: a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data batch processing method based on distributed scheduling as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 A flow chart of a data batch processing method based on distributed scheduling provided by a first embodiment of the present invention; Figure 2 A structural block diagram of a data batch processing system based on distributed scheduling provided by a second embodiment of the present invention; Figure 3 A schematic diagram of the hardware structure of a computer provided in the third embodiment of the present invention.

[0024] The embodiments of the present invention will be further described below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout are the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0026] In the description of the embodiments of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0027] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0028] Embodiment 1 In the first embodiment of the present invention, see Figure 1 , a data batch processing method based on distributed scheduling, comprising the following steps S01 to S07: S01, acquiring multi-source heterogeneous data, and preprocessing the multi-source heterogeneous data to form a standardized data set; In this embodiment, when preparing the data to be processed at the end of the day, the required data needs to be sorted and summarized. The multi-source heterogeneous data involved includes business system databases and SFTP (Simple File Transfer Protocol) data reconciliation files, log files, etc. Specifically, the sources of multi-source heterogeneous data include but are not limited to: Business databases: such as MySQL (a relational database management system), PostgreSQL (a powerful open source relational database system), MongoDB (a non-relational database, a document database), etc.

[0029] Log files: such as Web (World Wide Web) server logs, API (Application Programming Interface) access logs, etc.

[0030] Structured data files: such as CSV (comma-separated value file), JSON (JavaScript Object Notation, a lightweight data exchange format), Parquet (an efficient column storage format), Avro (a data serialization system) format files, etc.

[0031] Third-party data interface: such as financial risk control data API (application programming interface), etc.

[0032] Specifically, the step of preprocessing the multi-source heterogeneous data to form a standardized data set includes: Completing the missing values ​​of the multi-source heterogeneous data by using mean filling or interpolation methods; Normalizing the multi-source heterogeneous data by using standardized scaling or minimum and maximum scaling to fill missing values; Obvious abnormal data in the multi-source heterogeneous data after normalization processing is filtered out to form a standardized data set.

[0033] In this embodiment, after missing values ​​are filled in, multi-source heterogeneous data needs to be deduplicated. The main methods of data deduplication are: deduplication based on primary key, or using hash function to detect and remove duplicate data.

[0034] Then, the obvious abnormal data in the multi-source heterogeneous data should be filtered out, and then the data should be standardized to generate a standardized data set. Data standardization refers to the conversion of data from different sources into a unified format, such as formatting timestamps, unifying the units of amounts, and unifying text encodings.

[0035] S02, based on a predetermined first threshold, using a screening process to perform preliminary anomaly detection on the data set, and marking potential abnormal data; Specifically, the screening process includes: Arrange the data set in ascending order, and take the first data point in the sorted data set as the first judgment value ( ); The data point at the second position in the sorted data set is taken as the second judgment value ( ), a trend value is calculated based on the first judgment value and the second judgment value ; An upper threshold is calculated based on the first judgment value, the second judgment value and the trend value ( ) and the lower threshold ( ); Screening out potential abnormal data in the data set based on the upper threshold and the lower threshold using a comparison formula; The comparison formula includes:

[0036]

[0037] in, Represented as the first judgment value, Represented as the second judgment value, Represents potential abnormal data, Expressed as the coefficient.

[0038] In this embodiment, is 1.5, the first judgment value is the value at the 25% position in the data set, that is, the value at the 1 / 4 position after the data is sorted, and the second judgment value is the value at the 75% position in the data set, that is, the value at the 3 / 4 position after the data is sorted.

[0039] S03, for the remaining data set excluding the potential abnormal data, using an anomaly detection algorithm to calculate the anomaly score of each data point in the remaining data set; The anomaly detection algorithm includes: Randomly sub-sampling the remaining data set to construct multiple isolation trees; Calculating the path length of each data point in the remaining data set in each of the isolation trees, and calculating the average path length; The calculation formula is used to calculate based on the average path length to obtain an anomaly score.

[0040] Specifically, the calculation formula includes:

[0041] in, Represents data points The larger the value, the higher the abnormality. Represents the weight coefficient of the path length term (default value 1.0), which is used to adjust the contribution of path length to the anomaly score. Represents data points The average path length in all isolation trees. The shorter the path, the higher the probability of abnormality. Represents the dynamic normalization factor, which is related to the data set and the path length standard deviation Related, Represents the weight coefficient of the distribution density term (default value 0.5), which is used to adjust the contribution of density to the anomaly score. represents the slope adjustment factor (default value is 10), which controls the sensitivity of density differences. Represents data points The local density of (such as density estimation based on K nearest neighbors), the lower the density, the higher the possibility of abnormality. Represents the average density of the data set, which is used to standardize local density values;

[0042] represents a dynamic normalization factor that combines the harmonic number term and the path length standard deviation, Represents the mixing coefficient (default value 0.7), which is used to balance the weights of the harmonic number term and the standard deviation term. represents the harmonic number, defined as , for the path length criterion, where Indicates from 1 to The sum of the reciprocals of the natural numbers, Represents natural numbers The reciprocal of represents the number of data sets, It represents the standard deviation of the path lengths of all data points, reflecting the path length.

[0043] It is understandable that the higher the anomaly score, the easier it is to isolate the data point, that is, the stronger its anomaly. In each tree, the path length refers to the path length from the root node to the leaf node where the data point is located. The shorter the path, the easier it is to isolate the data point, that is, it is a potential anomaly. By randomly splitting multiple trees, the degree of anomaly of the point can be judged based on the path length of the data point in different trees. The average of the path lengths of multiple trees is used as the anomaly score of the point.

[0044] S04, according to a preset second threshold, filter out data points whose anomaly scores exceed the second threshold and mark them as highly abnormal data; In this embodiment, by setting a second threshold, it is possible to determine which data points belong to abnormal data. Typically, the second threshold is set between 0.6 and 0.7. Based on experience, a higher threshold means a stricter determination of abnormal data, while a lower threshold means a relatively loose determination criterion. For data points marked as abnormal, they are regarded as potential outliers or extreme values, which are likely to interfere with subsequent data analysis.

[0045] It can be understood that, with the help of step S03 and step S04, abnormal data that is not identified in the screening process can be screened out. For example, in a high-dimensional data set, although some data points are within the threshold range set by the screening process, they are actually still abnormal due to a specific feature combination. Through step S03 and step S04, these abnormal points can be identified.

[0046] Specifically, suppose that a set of transaction data is being analyzed, and this set of data contains multiple features, such as transaction amount, time, user ID, etc. Some transactions may not be judged as abnormal after the screening process. However, in the anomaly detection algorithm model (IsolationForest), these transactions may be identified as abnormal due to a certain combination of feature values. Therefore, combining the screening process with the anomaly detection algorithm can detect abnormal data more comprehensively. This combined method can not only effectively improve the accuracy of data cleaning, but also significantly reduce the analysis bias caused by data anomalies.

[0047] S05, merging the potential abnormal data and the highly abnormal data to generate abnormal data; In this embodiment, the specific steps of generating abnormal data are as follows: 1. Get a set: Get a set of potential abnormal data (C) and a set of highly abnormal data (D).

[0048] 2. Calculate the intersection and union: Calculate the intersection (C∩D) and union (C∪D) of set C and set D.

[0049] 3. Assign weights: For the intersection, that is, the anomalies detected by both the screening process and the anomaly detection algorithm, assign high anomaly weights.

[0050] 4. De-duplication: Duplicates need to be removed, i.e. the intersection of the two, to ensure that the outliers are not calculated multiple times. The generated outlier data is the union of the two.

[0051] Classification of abnormal data: 1. High-risk outliers: Points that are detected as anomalies in both the screening process and the anomaly detection algorithm.

[0052] 2. Low-risk outliers: Points that are only detected as anomalies by the screening process or anomaly detection algorithm.

[0053] 3. Critical anomaly point: The point where the anomaly score is close to the threshold and requires secondary confirmation by manual or business logic.

[0054] How to handle abnormal data: 1. Delete: For those abnormal data that are obviously wrong, delete them directly.

[0055] 2. Correction: For abnormal data caused by input errors, try to use methods such as averaging or interpolation to correct them.

[0056] 3. Marking: For abnormal points that may be questionable, mark them first and then hand them over to the business logic for further processing.

[0057] S06, splitting the data set excluding the abnormal data into multiple data blocks, and assigning the multiple data blocks to corresponding processing nodes for parallel processing based on a distributed task scheduling strategy; In this embodiment, in order to improve computing efficiency, a distributed task scheduling strategy is adopted to split the data into multiple subtasks and distribute them to computing nodes, as follows: 6.1 Data splitting strategy: Split by data size: For example, every 100MB of data is a subtask.

[0058] Split by business logic: for example, partition by fields such as user ID and order number.

[0059] Split by time window: Split data by time periods such as hours, days, or weeks.

[0060] 6.2 Task Scheduling Algorithm: Priority-based task scheduling: high-priority tasks are executed first, and low-priority tasks are executed when the system load is low.

[0061] Polling-based task allocation: Evenly allocates tasks to each computing node, which is suitable for load balancing scenarios.

[0062] Scheduling based on dynamic load awareness: Computing nodes periodically report their load status to the scheduling center so that task allocation can be dynamically adjusted.

[0063] Scheduling based on computing node type: Computing nodes with different performance (such as GPU / FPGA and low-power nodes) execute complex and simple tasks respectively.

[0064] 6.3 Task allocation method: Static allocation: The execution node of each task is determined before the task starts. It is suitable for scenarios with relatively stable data scale.

[0065] Dynamic allocation: Dynamically select processing nodes based on the real-time load of the nodes, suitable for large-scale concurrent computing tasks.

[0066] 6.4 Optimization strategies during task execution: Parallel computing and task splitting: Use multi-process or distributed computing frameworks to implement data computing.

[0067] Batch processing: Improve I / O read and write performance by processing in batches rather than row by row.

[0068] Vectorized computing: Use data processing and analysis libraries to reduce interpreter overhead and improve computing efficiency.

[0069] 6.5 Task failure handling mechanism: Automatic retry mechanism: supports up to 3 automatic retries when a task fails, and records the failure log.

[0070] Idempotence guarantee: ensures that even if a task is executed multiple times, it will not affect the final data consistency.

[0071] Rollback of failed tasks: When some calculations fail, the system supports rolling back to the last successful state.

[0072] 6.6 Data security and encryption measures: TLS transmission encryption: Use Transport Layer Security Protocol version 1.3 encryption technology to ensure the security of task scheduling and data transmission.

[0073] Data storage encryption: Sensitive data is stored using Advanced Encryption Standard 256-bit encryption.

[0074] Access Control: Limit user access rights to data through role-based access control mechanisms.

[0075] It can be seen that the design comprehensively considers efficiency, flexibility, security and error handling capabilities, and can be applied to a variety of different application scenarios and needs.

[0076] S07, summarizing the processing results of each processing node, and generating a data report to feed back to the user end or business system.

[0077] The following is a summary of the processing results of each processing node, as follows: 7.1 Summary of results: Purpose: Integrate the calculation results of each processing node to obtain the final data processing result. Ensure the integrity and consistency of the results, and properly solve the problems such as sorting and deduplication.

[0078] 7.2 Result data merging: There are two main methods used: MapReduce-based distributed aggregation: Applicable to massive data computing, usually implemented using computing frameworks such as Hadoop or Spark.

[0079] Merge based on database partition tables: suitable for the storage and merging of structured data. Commonly used databases include MySQL (a relational database management system) and PostgreSQL (a powerful open source relational database system).

[0080] 7.3 Result storage format: Choose the appropriate storage method based on the data type: Structured data: It is recommended to store it in a relational database, such as MySQL (a relational database management system), ClickHouse (a column storage database for online analytical processing), or TiDB (a distributed database that combines traditional relational databases and modern distributed storage technologies).

[0081] Semi-structured data: Suitable for storage in NoSQL (non-relational) databases, such as MongoDB (a document-based NoSQL database) or Elasticsearch (a document-based NoSQL database) or Elasticsearch (an open source search engine built on Lucene that can also be used as a NoSQL database).

[0082] Distributed file storage: For large-scale data sets, you can consider using storage systems such as HDFS (Hadoop Distributed File System), Ceph (a distributed storage system that provides high performance, high reliability, and high scalability), or S3 (Amazon's Simple Storage Service).

[0083] 7.4 Result Feedback Mechanism: Provides multiple ways for users or systems to obtain the final data processing results: Message queue notification: After the task is completed, a notification is sent through Kafka (a high-throughput, low-latency distributed messaging system) or RabbitMQ (a widely used open source message broker software) to facilitate real-time response.

[0084] The data API provides a query interface: it supports front-end applications or third-party systems to query processing results through the API (application programming interface), enhancing the accessibility and flexibility of the system.

[0085] Visual reports: Use ECharts (an open source data visualization library from Baidu) or Grafana (a cross-platform open source data visualization analysis tool) to visualize data, making it easier for users to understand and analyze data.

[0086] These mechanisms work together to ensure that the entire process, from the integration of computational results to the final feedback to users, is efficient and reliable. This not only meets the needs of data processing, but also improves the user experience, providing strong support for both data analysis and decision support.

[0087] When implementing distributed scheduling data batch processing, in addition to the core data processing and task scheduling mechanisms, special attention should be paid to the following key issues to ensure the efficiency, stability and reliability of the system: 1. Load balancing; The load conditions of each node in a distributed system may vary, so it is necessary to allocate tasks reasonably to ensure load balancing of the entire system. Through dynamic load sensing, priority-based or polling-based task scheduling, it is possible to effectively avoid the situation where some nodes are overloaded while other nodes are idle, thereby improving the overall efficiency and resource utilization of the system.

[0088] 2. Fault tolerance mechanism; Since nodes in a distributed system may encounter problems such as failures or network interruptions, it is necessary to establish an effective fault-tolerance mechanism. This includes but is not limited to automatic retry mechanisms, failed task rollbacks, and idempotency guarantees to ensure that even if errors or abnormalities occur, the system can still maintain stable operation and minimize the impact on the final result.

[0089] 3. Data consistency; In a distributed environment, data inconsistency may occur because data is distributed on multiple nodes. To ensure data consistency in the system, appropriate consistency models and protocols need to be adopted, such as two-phase commit, Paxos algorithm or Raft protocol. In addition, transaction management, version control and regular synchronization can also enhance data consistency.

[0090] 4. Monitoring and logging; In order to find and solve problems in a timely manner, distributed systems need to have comprehensive monitoring and logging functions. Real-time monitoring of the status of each node, resource usage, and task execution progress can help administrators quickly locate problems. At the same time, detailed log records not only help to analyze the cause of the failure afterwards, but also provide valuable data support for system optimization.

[0091] In the consumer finance business sector, a large amount of transaction data and user behavior log data are generated every day. In order to meet the needs of risk control and data analysis, these data need to be summarized, processed and counted in real time to generate various user behavior indicators and business occurrence reports for business operations. By adopting distributed scheduling data batch processing technology, large tasks that originally took a long time can be decomposed into multiple small tasks, and the advantages of distributed computing can be used to efficiently complete data processing tasks within the specified time limit. In addition, this technology ensures the traceability and monitorability of the entire data flow, so that problems can be quickly intervened and solved when they arise, thereby ensuring the stable operation of the system and the accuracy of data processing. This method not only improves processing efficiency, but also enhances the adaptability to complex data environments and risk management levels. The present invention can be used in the field of financial products in the guarantee industry. The amount of data of financial products is huge and the real-time requirements are high. Distributed scheduling data batch processing technology can help institutions process a large amount of transaction data and user data, and provide real-time risk control and data analysis services.

[0092] In summary, a data batch processing method based on distributed scheduling has the following effects: 1. High throughput; Distributed scheduling can process multiple batches of data at the same time, significantly improving the throughput of data processing and thus speeding up data processing. This parallel processing capability enables the system to process large amounts of data in a short period of time.

[0093] 2. Scalability; The distributed scheduling system can flexibly increase or decrease processing nodes according to actual needs, and easily expand the system's processing capacity. This provides convenience for dealing with the ever-increasing amount of data and ensures that the system can continue to operate efficiently.

[0094] 3. High availability; In a distributed scheduling system, each node can back up each other. When some nodes fail, other nodes can continue to complete data processing tasks, ensuring the stability and high availability of the system and reducing service interruptions caused by single point failures.

[0095] 4. High resource utilization; The distributed scheduling system can distribute tasks evenly, making full use of the resources of each processing node and improving the overall resource utilization. Through the dynamic load balancing strategy, the system can optimize resource allocation and avoid resource waste.

[0096] 5. Flexibility; The distributed scheduling system supports flexible adjustment of data processing task priorities and allocation strategies to meet data processing requirements in different scenarios. Whether it is real-time data analysis or historical data processing, the system can adapt according to business needs.

[0097] 6. Easy to maintain and debug; The nodes in the distributed scheduling system can run independently, which is convenient for maintenance and debugging. In addition, the system is easy to monitor and manage logs, which helps to find and solve problems in time, improving the maintainability and stability of the system.

[0098] In summary, distributed scheduling data batch processing technology not only improves the speed and efficiency of data processing, but also enhances the flexibility, reliability and scalability of the system, and better meets the needs of large-scale data processing. These features make it an ideal choice for modern data-intensive applications.

[0099] Embodiment 2 like Figure 2 As shown, in a second embodiment of the present invention, a data batch processing system based on distributed scheduling is provided, and the system includes: The processing module 10 is used to obtain multi-source heterogeneous data and pre-process the multi-source heterogeneous data to form a standardized data set; A detection module 20, configured to perform preliminary anomaly detection on the data set based on a predetermined first threshold value and to mark potential abnormal data by using a screening process; A calculation module 30 is used to calculate the abnormal score of each data point in the remaining data set excluding the potential abnormal data by using an abnormality detection algorithm; A screening module 40 is used to screen out data points whose abnormality scores exceed a preset second threshold value according to the second threshold value, and mark them as highly abnormal data; A merging module 50, configured to merge the potential abnormal data and the highly abnormal data to generate abnormal data; A parallel module 60, configured to split the data set excluding the abnormal data into multiple data blocks, and assign the multiple data blocks to corresponding processing nodes for parallel processing based on a distributed task scheduling strategy; The feedback module 70 is used to summarize the processing results of each processing node and generate a data report to feed back to the user end or business system.

[0100] The data batch processing system based on distributed scheduling provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment.

[0101] Embodiment 3 like Figure 3As shown, in the third embodiment of the present invention, the embodiment of the present invention provides the following technical solution, a computer, including a memory 202, a processor 201, and a computer program stored in the memory 202 and executable on the processor 201, and the processor 201 implements the data batch processing method based on distributed scheduling as described above when executing the computer program.

[0102] Specifically, the processor 201 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0103] Among them, the memory 202 may include a large-capacity memory for data or instructions. For example, but not limitation, the memory 202 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 202 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 202 may be inside or outside the data processing device. In a specific embodiment, the memory 202 is a non-volatile memory. In a specific embodiment, the memory 202 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (Static Random-Access Memory, referred to as SRAM) or a dynamic random access memory (Dynamic Random Access Memory, referred to as DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (Fast Page Mode Dynamic Random Access Memory, referred to as FPMDRAM), an extended data output dynamic random access memory (Extended Date Out Dynamic Random Access Memory, referred to as EDODRAM), a synchronous dynamic random access memory (Synchronous Dynamic Random-Access Memory, referred to as SDRAM), etc.

[0104] The memory 202 may be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 201 .

[0105] The processor 201 implements the above-mentioned data batch processing method based on distributed scheduling by reading and executing the computer program instructions stored in the memory 202.

[0106] In some embodiments, the computer may further include a communication interface 203 and a bus 200. Figure 3 As shown, the processor 201, the memory 202, and the communication interface 203 are connected via a bus 200 and communicate with each other.

[0107] The communication interface 203 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 203 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0108] The bus 200 includes hardware, software, or both, and couples the components of the computer to each other. The bus 200 includes, but is not limited to, at least one of the following: a data bus (DataBus), an address bus (AddressBus), a control bus (ControlBus), an expansion bus (ExpansionBus), and a local bus (LocalBus). By way of example and not limitation, the bus 200 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, or a Low Pin Count (LPC) bus. Bus, memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local Bus (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, bus 200 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application considers any suitable bus or interconnect.

[0109] Embodiment 4 In a fourth embodiment of the present invention, in combination with the above-mentioned data batch processing method based on distributed scheduling, the embodiment of the present invention provides the following technical solution, a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned data batch processing method based on distributed scheduling is implemented.

[0110] Those skilled in the art will appreciate that the logic and / or steps described in the flowchart as data or in other ways herein, for example, can be considered as a sequenced data table of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses.

[0111] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0112] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data batch processing method based on distributed scheduling, characterized in that: The method comprises: Acquire multi-source heterogeneous data, and pre-process the multi-source heterogeneous data to form a standardized data set; Based on a predetermined first threshold, using a screening process to perform preliminary anomaly detection on the data set to mark potential abnormal data; For the remaining data set excluding the potential abnormal data, an anomaly detection algorithm is used to calculate the anomaly score of each data point in the remaining data set; According to a preset second threshold, data points whose anomaly scores exceed the second threshold are screened out and marked as highly abnormal data; Combining the potential abnormal data and the highly abnormal data to generate abnormal data; Splitting the data set excluding the abnormal data into multiple data blocks, and assigning the multiple data blocks to corresponding processing nodes for parallel processing based on a distributed task scheduling strategy; The processing results of each processing node are summarized and a data report is generated and fed back to the user end or business system.

2. The data batch processing method based on distributed scheduling according to claim 1 is characterized in that: The multi-source heterogeneous data includes business system databases and log files.

3. The data batch processing method based on distributed scheduling according to claim 1 is characterized in that: The step of preprocessing the multi-source heterogeneous data to form a standardized data set includes: Completing the missing values ​​of the multi-source heterogeneous data by using mean filling or interpolation methods; Normalizing the multi-source heterogeneous data by using standardized scaling or minimum and maximum scaling to fill missing values; Obvious abnormal data in the multi-source heterogeneous data after normalization processing is filtered out to form a standardized data set.

4. The data batch processing method based on distributed scheduling according to claim 1 is characterized in that: The screening process includes: Arrange the data set in ascending order, and take the data point at the first position in the sorted data set as the first judgment value; taking a data point at a second position in the sorted data set as a second judgment value, and calculating a trend value based on the first judgment value and the second judgment value; An upper threshold and a lower threshold are calculated based on the first judgment value, the second judgment value and the trend value; Potential abnormal data in the data set is screened out based on the upper threshold and the lower threshold using a comparison formula.

5. The data batch processing method based on distributed scheduling according to claim 4 is characterized in that: The comparison formula includes: in, Represented as the first judgment value, Represented as the second judgment value, Represents potential abnormal data, Expressed as the coefficient.

6. The data batch processing method based on distributed scheduling according to claim 1 is characterized in that: The anomaly detection algorithm includes: Randomly sub-sampling the remaining data set to construct multiple isolation trees; Calculating the path length of each data point in the remaining data set in each of the isolation trees, and calculating the average path length; The calculation formula is used to calculate based on the average path length to obtain an anomaly score.

7. The data batch processing method based on distributed scheduling according to claim 6 is characterized in that: The calculation formula includes: in, Represents data points The anomaly score, represents the weight coefficient of the path length term, Represents data points The average path length among all isolation trees, represents the dynamic normalization factor, represents the weight coefficient of the distribution density term, represents the slope adjustment factor, Represents data points The local density of Represents the average density of the data set; represents the dynamic normalization factor, represents the mixing coefficient, represents the harmonic number, represents the number of data sets, Represents the standard deviation of the path lengths of all data points.

8. A data batch processing system based on distributed scheduling, characterized in that: The system comprises: A processing module, used for acquiring multi-source heterogeneous data and preprocessing the multi-source heterogeneous data to form a standardized data set; A detection module, configured to perform preliminary anomaly detection on the data set based on a predetermined first threshold value and mark potential abnormal data by using a screening process; A calculation module, configured to calculate an abnormality score of each data point in the remaining data set by using an abnormality detection algorithm for the remaining data set excluding the potential abnormal data; A screening module, configured to screen out data points whose anomaly scores exceed a preset second threshold value according to the second threshold value, and mark them as highly abnormal data; A merging module, used for merging the potential abnormal data and the highly abnormal data to generate abnormal data; A parallel module, used for splitting the data set excluding the abnormal data into multiple data blocks, and allocating the multiple data blocks to corresponding processing nodes for parallel processing based on a distributed task scheduling strategy; The feedback module is used to summarize the processing results of each processing node and generate a data report to feed back to the user end or business system.

9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data batch processing method based on distributed scheduling is implemented as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the data batch processing method based on distributed scheduling as described in any one of claims 1 to 7 is implemented.

Images