A method and system for processing massive data based on heterogeneous data sources

Abstract

The application relates to a massive data processing method and system based on heterogeneous data sources. The method comprises the following steps: writing service calling data into storage, querying service calling data, cleaning data files at a fixed time, and intelligently warning through a shell script. The method can support rapid expansion of a service system, greatly improve the interface concurrency, support interface daily call of the order of 100 million, and be used for storage and calling of massive data; the method supports system expansion without awareness of external users, and the interface can be normally called during expansion. The method adopts multi-data source engine cooperation query, supports cross-data source query, is suitable for various query scenes, improves data real-time query capability and multidimensional analysis capability, the method can intelligently clean useless storage data and files in time and quickly, manual patrol cleaning of operation and maintenance is not needed, and the labor and machine costs are reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mixed data source data processing methods, and particularly relates to a massive data processing method and system based on heterogeneous data sources. BACKGROUND

[0002] Many application systems need to obtain information such as the number of published services, the interface failure rate, the interface calling time, the interface calling quantity (day, week, month, year), the number of supported applications and the like during the working process, for example, when a data service system is used to count the interface calling conditions. At present, most of the existing systems adopt the architecture scheme of mongodb+mysql+operation manual data cleaning. The mongodb is used to store calling data, the mongodb data is processed by a timing task to form statistical data, and then the statistical data is stored in a single table of the mysql. The operation is manually cleaned once a year to remove useless data files in the mongodb.

[0003] However, if the calling interface users increase dramatically due to a sudden event, the service daily calling quantity reaches the level of ten million or one billion, and the single-second single-interface concurrency reaches several thousand (such as the service interface of providing personal information query in a city with ten million population), the above-mentioned scheme based on mongodb+mysql+operation manual data cleaning will have the following defects:

[0004] 1. The daily calling quantity of the interface in the above-mentioned scheme can only support the level of ten thousand, and it is impossible to support the situation that the calling quantity reaches the level of one billion per day due to a sudden event.

[0005] 2. The mongodb stores data files, and it is difficult to expand the capacity. The expansion of the mongo cluster is complex, the service will be stopped during the expansion process, the cost of machines and labor is huge, and when the concurrency suddenly increases, it is difficult to support the expansion in a short time in a way that is not perceived by the user.

[0006] 3. The interface calling statistical data is stored in a single table of the mysql. If 10,000 interfaces are opened to the outside, more than 130,000 data will be generated per day, 400,000 data per month, and 5,000,000 data per year. When the data quantity of the single table of the mysql exceeds 5,000,000, the disk io bottleneck will be touched, which will cause the query to be slow, the efficiency to be low, the interface calling to be timed out, and even the interface calling to be unavailable.

[0007] 4. The storage middleware based on the mysql and the mongodb does not support cross-data source query, is not friendly to complex query and statistics, and the query efficiency is difficult to meet the complex OLAP demand.

[0008] 5. Manually clean up data files. For daily data file sizes below 1GB, a typical file server with a 1TB disk size only needs to manually clean up data once a year. However, for daily data volumes exceeding 50GB, the data disk will fill up in just over ten days, requiring frequent disk checks and manual file cleanup by maintenance personnel, consuming labor costs. If cleanup is not timely, it can also lead to service unavailability. Summary of the Invention

[0009] Terminology Explanation

[0010] Presto: Presto is an open-source, fully in-memory parallel computing-based distributed SQL query engine from Facebook, suitable for interactive analytical queries.

[0011] Catalog: A catalog is a data source. Each data source connection has a name, and a catalog can contain multiple schemas. Data sources are referenced through connectors, and the `show catalogs` command can be used to see all data sources that Presto has connected to.

[0012] A schema is like a database; a schema contains multiple tables. The command `show schemasfrom 'catalog_name'` lists all schemas under a catalog.

[0013] Kafka: Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action streams of data from consumers on a website.

[0014] To overcome the shortcomings of existing MongoDB + MySQL + manual data clearing architecture solutions, this invention proposes a novel method for processing massive amounts of data based on heterogeneous data sources. This invention aims to solve the following problems:

[0015] 1. In response to situations where sudden events cause the daily service call volume of a single interface to reach tens of millions or even hundreds of millions, a data storage solution that is easy to expand is proposed to solve the problem of expansion difficulties.

[0016] 2. Optimize the storage solution to resolve the issue of database I / O blocking and database freezing that causes the service platform to become unavailable when the data volume of a single MySQL table exceeds 50 million records.

[0017] 3. Improve the speed and efficiency of single-table queries.

[0018] 4. Enables cross-data source queries for multiple data sources, meeting complex OLAP requirements.

[0019] 5. Propose automatic cleaning rules and algorithms to achieve intelligent data file cleaning, replacing manual data cleaning and saving labor costs.

[0020] Overall, the present invention achieves the above objectives through the following technical concepts and strategies:

[0021] (1) Efficient expansion solution: Replace the main data storage engine, use MongoDB only as a temporary storage middleware and store the final data in ElasticaSearch. Since ElasticaSearch expansion is achieved by adding machine nodes in the cluster, it is easier to expand.

[0022] (2) MySQL intelligently partitions tables according to specific rules and algorithms.

[0023] (3) In order to realize various queries such as single table query and complex OLAP, multiple data source engines (mysql, mongodb, ElasticaSearch, redis) are used to store data. Multiple data sources cooperate to complete the query, improve query efficiency and share the query pressure.

[0024] (4) Introducing the Presto engine to support cross-data source query analysis of MySQL and ElasticaSearch.

[0025] (5) Multiple data cleaning solutions such as scheduled tasks, shell scripts, and database scheduled tasks are available to clean up MongoDB, MySQL statistical tables, and ElasticaSearch on a scheduled basis.

[0026] (6) Introduce intelligent early warning. After the above solutions are implemented, if the system still has unmanageable server pressure or database pressure, it will conduct daily inspections and intelligently send early warning messages to the administrator.

[0027] Specifically, this invention provides a method for processing massive amounts of data based on heterogeneous data sources, the method comprising:

[0028] S1. Service call data is written to storage, including MongoDB and Elasticsearch;

[0029] S2. Query service call data;

[0030] S3. Regularly clean up data files;

[0031] S4. Intelligent alerts via shell scripts.

[0032] Furthermore, according to some embodiments of the present invention, the service call data in step S1 of the massive data processing method based on heterogeneous data sources of the present invention includes a JSON string recording API service call time, apiId, application name, parameters, time consumption, error information, and error code information.

[0033] Furthermore, the service call data writing and storage step S1 in the massive data processing method based on heterogeneous data sources of the present invention includes:

[0034] S11. Asynchronously write the specific information of the service call data to MongoDB for data statistics and querying, and create a new collection for MongoDB every day. Start a scheduled task at 23:59 every day to create a new MongoDB table and write the data of the day to the data table of the day.

[0035] S12. Asynchronously write the specific information of the service call data to Elasticsearch for user queries, and create new indexes for Elasticsearch on a rolling basis every month (create templates, and generate new indexes monthly based on templates and creation strategies).

[0036] S13. Every 10 seconds, perform statistics on the number of API calls, average execution time, number of failures, and number of calls for MongoDB data, categorized by year, month, day, and hour, and save the statistical data to a MySQL statistics table;

[0037] S14. Create a new data statistics table for MySQL every year. Create a new table at the end of each year. Synchronize the data of the previous month's table with the data of the next month's table. Create a new data table named dytj_current date year at 23:00:00 on the last day of each year. At the same time, synchronize the data of December to the new data table. On February 1st of each year, clean up the data of December of the previous year in the new table.

[0038] Furthermore, according to some embodiments of the present invention, when the system experiences a sudden surge in database and / or server pressure and insufficient resource space during operation, the massive data processing method based on heterogeneous data sources of the present invention expands its capacity in the following manner:

[0039] (1) For MySQL, when there is insufficient resource space, add a machine to MySQL and store historical data and current data in a separate database using the MySQL table partitioning strategy;

[0040] (2) For MongoDB, MongoDB only stores temporary data as a temporary storage middleware. When there is insufficient resource space, only one day's data is retained and the middleware data is deleted.

[0041] (3) For Elasticsearch, assess the server resources that need to be added, calculate the number of machines that need to be scaled up based on the call volume, and perform horizontal scaling.

[0042] (4) For application servers, assess the server resources that need to be increased based on the total concurrency and the concurrency supported by a single server configuration, and perform horizontal scaling.

[0043] Furthermore, the query service call data in step S2 of the massive data processing method based on heterogeneous data sources of the present invention includes:

[0044] (1) Single table query

[0045] ① Query API service call statistics. By querying the MySQL statistics table, obtain statistical data information in the dimensions of year, month, day, and hour.

[0046] ② Query API service call data information. By querying Elasticsearch, you can obtain data information such as API service name, call time, application name, data content, parameters, time consumption, and error code.

[0047] (2) Cross-data source query

[0048] ① Configure the Presto catalog. Locate the catalog directory in the Presto installation directory, and then add connector files in the catalog directory. Create mysql.properties, mongodb.properties, and Elasticsearch.properties, and configure the connector information (address, username, password, etc.).

[0049] ② Write SQL for mixed queries. The method for writing mixed query SQL is the same as that for ordinary SQL. The table name for the query is written as catalog.schame.tableName (catalog is the name of properties), which completes the cross-data source query.

[0050] Furthermore, in the above-mentioned method for processing massive amounts of data based on heterogeneous data sources, the query API service call data information in the single-table query also includes aggregation analysis based on the data information obtained from the query;

[0051] The aggregate analysis includes error data distribution analysis, error type distribution analysis, and analysis of the time period with the most errors.

[0052] Furthermore, the timed data file cleaning step S3 of the massive data processing method based on heterogeneous data sources of the present invention includes:

[0053] (1) MongoDB data cleanup

[0054] Set up a daily data cleanup task to delete MongoDB data that is valid for 2 days. Delete the MongoDB data from the previous 2 days each day.

[0055] (2) Elasticsearch data cleaning

[0056] ① When creating an index in Elasticsearch, a scheduled data deletion task is set up synchronously. The data is valid for one year and will be automatically deleted upon expiration.

[0057] ② Manually write a shell script for detection and put it on the server. Check the disk space daily. When the disk space is less than 10%, delete files according to importance and index creation time strategy until the disk space reaches 80%.

[0058] (3) MySQL data cleanup

[0059] ① Set up a scheduled data deletion task to be cleaned up annually. The data is valid for 3 years and the statistical table of data older than 3 years will be automatically deleted when it expires.

[0060] ② Manually write a shell script for detection and put it on the server. Check the disk space daily. When the disk space is less than 10%, delete files according to importance and data storage time strategy until the disk space is more than 10%.

[0061] ③ For MySQL database creation events, check the data volume of a single table daily. If the data volume of a single table exceeds 100 million, write data to the alarm table and notify the platform maintenance personnel to check and handle it.

[0062] Furthermore, the intelligent early warning via shell script in step S4 of the massive data processing method based on heterogeneous data sources of the present invention includes:

[0063] S41. Write an intelligent early warning shell script, and then create the xtyj.sh file in / home / opt;

[0064] S42. Query disk space. If the disk space utilization rate reaches 85%, send an XT_YJ_TOPIC message to Kafka. The message content includes the server IP, CPU and disk usage.

[0065] S43. Set a scheduled task for the intelligent early warning shell script to execute the script at 1:00 every day;

[0066] S44. The backend code listens for messages in the corresponding Kafka topic XT_YJ_TOPIC, and after receiving the message, sends an alert message to the system administrator;

[0067] S45. After receiving the warning message, the system administrator checks for server anomalies and takes appropriate action.

[0068] If, after implementing the above solutions, the system still experiences unmanageable server or database pressure, it will conduct daily inspections and intelligently send alert messages to the system administrator.

[0069] Secondly, the present invention also provides a massive data processing system based on heterogeneous data sources, the processing system comprising:

[0070] The storage module is used to write service call data to storage, which includes MongoDB and Elasticsearch;

[0071] The query module is used to query service call data;

[0072] The cleanup module is used to clean up data files on a regular schedule.

[0073] The early warning module is used to provide intelligent early warnings via shell scripts.

[0074] In addition, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for processing massive amounts of data based on heterogeneous data sources.

[0075] In summary, the massive data processing method based on heterogeneous data sources of this invention has the following advantages:

[0076] (1) The method of the present invention can support rapid expansion of the service system, greatly increase the interface concurrency, support the daily hundreds of millions of interface calls, and can be used for the storage and calling of massive data; and supports system expansion without external users' awareness, and the interface can be called normally during the expansion period.

[0077] (2) The method of the present invention adopts multi-data source engine collaborative query, supports cross-data source query, is applicable to various query scenarios, and improves the real-time data query capability and multi-dimensional analysis capability.

[0078] (3) The method of the present invention can intelligently clean up useless storage data and files in a timely and rapid manner, without the need for manual inspection and cleaning by maintenance personnel, thus reducing labor and machine costs.

[0079] (4) The method of the present invention provides an intelligent early warning function. When the pressure on the database and server suddenly increases, it can send an early warning to the platform maintenance personnel via email, SMS or other means. Attached Figure Description

[0080] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1 This is a flowchart illustrating the overall implementation process of the method of the present invention.

[0082] Figure 2 This is a block diagram of the service call data storage scheme in the method of the present invention.

[0083] Figure 3 This is a flowchart of the service call data storage process in the method of the present invention.

[0084] Figure 4 This is a flowchart of the service call data query scheme in the method of the present invention.

[0085] Figure 5 This is a block diagram of the data periodic cleanup scheme in the method of the present invention.

[0086] Figure 6 This is a flowchart of the intelligent early warning process in the method of the present invention.

[0087] Figure 7 This is a schematic diagram of the composition structure of the data processing system of the present invention. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. This invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this invention.

[0089] At the same time, it should be understood that the scope of protection of the present invention is not limited to the specific embodiments described below; it should also be understood that the terminology used in the embodiments of the present invention is for describing specific embodiments and not for limiting the scope of protection of the present invention.

[0090] Example: A method for processing massive amounts of data based on heterogeneous data sources

[0091] like Figure 1 As shown, this method includes the following steps:

[0092] S1. Service call data is written to storage.

[0093] Assumption background:

[0094] The service platform provides the following number of interfaces, divided into high-concurrency call interfaces and low-concurrency call interfaces:

[0095]

[0096] MySQL data volume:

[0097] The number of statistical data items generated (per day) = total number of services * 24 * 1.5 ≈ 110,000 items;

[0098] The number of statistical data generated (data items per year) = the number of statistical data generated (data items per day) * 365 ≈ 40 million data items.

[0099] MongoDB data volume:

[0100] Detailed data volume (daily) = Number of high-concurrency interfaces * Average daily call volume * Data volume + Number of low-concurrency interfaces * Average daily call volume * Data volume

[0101] =5000w*10*1kb + 1w*3000*1kb

[0102] ≈500G.

[0103] ElasticaSearch data volume:

[0104] Monthly data volume = Daily data volume * 30 ≈ 15T;

[0105] Data volume per quarter = Data volume (per day) * 90 ≈ 45T;

[0106] Annual data volume = Daily data volume * 365 ≈ 180T.

[0107] like Figure 2 and Figure 3 As shown, the service call data storage process includes the following steps:

[0108] S11. Service call data is asynchronously written to MongoDB.

[0109] Service call data, such as API service call time, apiId, application name, parameters, time consumption, error information, error code information, and other detailed information recorded in JSON format, is asynchronously written to MongoDB for data statistics and queries.

[0110] MongoDB table partitioning by day

[0111] Create a new collection for MongoDB every day. Start a scheduled task at 23:59 every day to create a new MongoDB table. Write the data of the day into the table for the day and only retain the data of the past two days.

[0112] S12. Service call data is asynchronously written to Elasticsearch.

[0113] The specific information of the service call data mentioned above is asynchronously written to Elasticsearch for user queries.

[0114] Elasticsearch generates new indexes monthly.

[0115] Create templates and generate new indexes for Elasticsearch on a rolling basis each month based on the templates and strategies.

[0116] S13. Save statistical data to MySQL

[0117] Every 10 seconds, the MongoDB data is analyzed and statistically analyzed by year, month, day, and hour for API call volume, average execution time, number of failures, and number of calls. The statistical data is then saved to a MySQL statistics table.

[0118] (1) Sync statistical data by hour using insertAndUpdateHourData;

[0119] (2) Synchronize statistical data by day using insertAndUpdateDayData;

[0120] (3) Synchronize monthly statistical data using insertAndUpdateMonthData;

[0121] (4) Statistical data is synchronized by year using insertAndUpdateYearData.

[0122] S14. MySQL statistical tables broken down by year

[0123] A new data table is created for MySQL every year. A scheduled task at the end of each year creates the new table, synchronizes the previous month's data with the new table, and writes the next month's data into the new table. A new data table named `dytj_current date_year` is created at 23:00:00 on the last day of each year, and the December data is synchronized to the new table. On February 1st each year, a scheduled task cleans up the previous year's December data in the new table.

[0124] Expansion

[0125] During system operation, after implementing the data file cleanup solution, if the database or server pressure suddenly surges and resource space becomes insufficient, the intelligent early warning function will send alerts to platform maintenance personnel via email, SMS, etc., indicating which server and application are experiencing resource shortages. Different solutions will be implemented for scaling up or handling MySQL, MongoDB, Elasticsearch, and application servers.

[0126] (1) mysql

[0127] For MySQL, a MySQL table partitioning strategy is used to add machines to the MySQL database when resource space is insufficient, and to partition the database to store historical data and current data.

[0128] (2) MongoDB

[0129] For MongoDB, MongoDB only stores temporary data and acts as a temporary storage middleware. When there is insufficient resource space, it only retains one day's worth of data and deletes other data in the middleware table.

[0130] (3) Elasticsearch

[0131] For Elasticsearch, assess the required additional server resources and perform horizontal scaling, calculating the number of machines needed for expansion based on call volume:

[0132] Daily data volume = ∑ Daily call volume of statistical form service * 1K;

[0133] If you need to retain one year's worth of data, then the data storage capacity is:

[0134] Annual data volume = Daily data volume * 365.

[0135] (4) Application Server

[0136] For application servers, assess the required additional server resources based on the total concurrency and the concurrency supported by the single server configuration, and then perform horizontal scaling.

[0137] Total concurrency (per second) = Hourly call count stored in the statistics table / 3600.

[0138] S2. Query service call data

[0139] like Figure 4 As shown, service call data query includes single table query and cross data source query, and the specific process is as follows:

[0140] (I) Single table query (simple query)

[0141] (1) Query API service call statistics. By querying the MySQL statistics table, you can directly obtain statistical data information in the dimensions of year, month, day and hour.

[0142] (2) Query API service call data information. By querying Elasticsearch, obtain specific data information such as API service name, call time, application name, data content, parameters, time consumption, and error code.

[0143] It also supports aggregation analysis based on the data obtained from the query, such as statistical information analysis of error data distribution, error type distribution, and the time period with the most errors.

[0144] (ii) Cross-data source queries (complex queries)

[0145] For complex business scenarios, queries often require joint queries using MySQL, Elasticsearch, and MongoDB. Presto enables cross-data source queries.

[0146] (1) Configure the Presto catalog. Locate the catalog directory in the Presto installation directory, and then add connector files in the catalog directory. Create mysql.properties, mongodb.properties, and Elasticsearch.properties, and configure the connector information, including address, username, password, etc.

[0147] (2) Write SQL for mixed queries to complete cross-data source queries. The method for writing mixed query SQL is the same as ordinary SQL. The table name to be queried is catalog.scheme.tableName, where catalog is the name of the properties, as shown in the table below:

[0148] Data source type catalog Table name mysql mysql mysql.database.table mongodb mongodb mongodb.database.table Elasticsearch Elasticsearch Elasticsearch.database.table

[0149] S3. Regularly clean up data files

[0150] like Figure 5 As shown, data file cleaning includes the following process:

[0151] (1) MongoDB data cleanup

[0152] Set up a daily data cleanup task with a two-day validity period. Delete the MongoDB data from the previous two days each day (collection name: dyrz_collection of the previous two days' dates).

[0153] (2) Elasticsearch data cleaning

[0154] ① When creating an index in Elasticsearch, a scheduled data deletion task is set up synchronously. The data is valid for one year and will be automatically deleted upon expiration.

[0155] ② Manually write a shell script for detection and put it on the server. Check the disk space daily. When the disk space is less than 10%, delete files according to importance and index creation time strategy until the disk space reaches 80%.

[0156] (3) MySQL data cleanup

[0157] ① Set up a scheduled data deletion task to be cleaned up annually. The data is valid for 3 years and the statistical table of data older than 3 years will be automatically deleted when it expires.

[0158] ② Manually write a shell script for detection and put it on the server. Check the disk space daily. When the disk space is less than 10%, delete files according to importance and data storage time strategy until the disk space is more than 10%.

[0159] ③ For MySQL database creation events, check the data volume of a single table daily. If the data volume of a single table exceeds 100 million, write data to the alarm table and notify the platform maintenance personnel via email and SMS to check. If any abnormalities are found, take appropriate measures to handle them.

[0160] S4. Intelligent alerts via shell scripts

[0161] like Figure 6 As shown, intelligent early warning includes the following process:

[0162] S41. Write an intelligent early warning shell script, and then create the xtyj.sh file in / home / opt;

[0163] S42. Query disk space. If the disk space utilization rate reaches 85%, send an XT_YJ_TOPIC message to Kafka. The message content includes the server IP, CPU and disk usage.

[0164] S43. Set a scheduled task for the intelligent early warning shell script to execute the script daily at 1:00 AM.

[0165] 0 1*** / home / opt / xtyj.sh;

[0166] S44. The backend code listens for messages in the corresponding Kafka topic XT_YJ_TOPIC. After receiving the message, it sends an alert message to the system administrator via email, SMS, etc.

[0167] S45. After receiving the warning message, the system administrator checks for server anomalies and takes appropriate action.

[0168] A massive data processing system based on heterogeneous data sources, such as Figure 7 As shown, it includes:

[0169] The storage module is used to write service call data to storage, which includes MongoDB and Elasticsearch;

[0170] The query module is used to query service call data;

[0171] The cleanup module is used to clean up data files on a regular schedule.

[0172] The early warning module is used to provide intelligent early warnings via shell scripts.

[0173] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, substitutions, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for processing massive amounts of data based on heterogeneous data sources, characterized in that, The method includes: S1. Service call data is written to storage; the storage includes MongoDB and Elasticsearch; the service call data includes API service call time, apiId, application name, parameters, time consumption, error information, and error code information recorded in JSON format; the writing of service call data to storage includes: S11. Asynchronously write the specific information of the service call data to MongoDB for data statistics queries, and create a new collection for MongoDB every day; S12. Asynchronously write the specific information of the service call data to Elasticsearch for user queries, and create new indexes for Elasticsearch on a rolling basis every month; S13. Every 10 seconds, perform statistics on the number of API calls, average execution time, number of failures, and number of calls for MongoDB data, categorized by year, month, day, and hour, and save the statistical data to a MySQL statistics table; S14. Create new data tables for MySQL every year; S2. Query service call data; S3. Regularly clean up data files; S4. Intelligent alerts via shell scripts, including: S41. Write an intelligent early warning shell script, and then create the xtyj.sh file in / home / opt; S42. Query disk space. If the disk space utilization rate reaches 85%, send an XT_YJ_TOPIC message to Kafka. The message content includes the server IP, CPU and disk usage. S43. Set a scheduled task for the intelligent early warning shell script to execute the script at 1:00 every day; S44. The backend code listens for messages in the corresponding Kafka topic XT_YJ_TOPIC, and after receiving the message, sends an alert message to the system administrator; S45. After receiving the warning message, the system administrator checks for server anomalies and takes appropriate action.

2. The method for processing massive amounts of data based on heterogeneous data sources according to claim 1, characterized in that, When the system experiences a sudden surge in database and / or server pressure and insufficient resource space during operation, the following methods can be used to expand capacity: (1) For MySQL, when there is insufficient resource space, add a machine to MySQL and store historical data and current data in a separate database using the MySQL table partitioning strategy; (2) For MongoDB, MongoDB only stores temporary data as a temporary storage middleware. When there is insufficient resource space, it retains only 1 day of data and deletes the data in the middle table. (3) For Elasticsearch, assess the server resources that need to be added, calculate the number of machines that need to be scaled up based on the call volume, and perform horizontal scaling. (4) For application servers, assess the server resources that need to be increased based on the total concurrency and the concurrency supported by a single server configuration, and perform horizontal scaling.

3. The method for processing massive amounts of data based on heterogeneous data sources according to claim 1, characterized in that, The query service call data mentioned in step S2 includes: (1) Single table query ① Query API service call statistics. By querying the MySQL statistics table, obtain statistical data information in the dimensions of year, month, day, and hour. ② Query API service call data information. By querying Elasticsearch, you can obtain data information such as API service name, call time, application name, data content, parameters, time consumption, and error code. (2) Cross-data source query ① Configure the Presto catalog. Locate the catalog directory in the Presto installation directory, and then add connector files in the catalog directory. Create mysql.properties, mongodb.properties, and Elasticsearch.properties, and configure the connector information. ② Write a mixed query SQL, with the table name to be queried being catalog.schame.tableName, to complete the cross-data source query.

4. The method for processing massive amounts of data based on heterogeneous data sources according to claim 3, characterized in that, The query API service call data information mentioned in the single table query also includes aggregation analysis based on the data information obtained from the query; The aggregate analysis includes error data distribution analysis, error type distribution analysis, and analysis of the time period with the most errors.

5. The method for processing massive amounts of data based on heterogeneous data sources according to claim 1, characterized in that, The timed data file cleanup mentioned in step S3 includes: (1) MongoDB data cleanup Set up a daily data cleanup task to delete MongoDB data that is valid for 2 days. Delete the MongoDB data from the previous 2 days each day. (2) Elasticsearch data cleaning ① When creating an index in Elasticsearch, a scheduled data deletion task is set up synchronously. The data is valid for one year and will be automatically deleted upon expiration. ② Manually write a shell script for detection and put it on the server. Check the disk space daily. When the disk space is less than 10%, delete files according to importance and index creation time strategy until the disk space is 80%. (3) MySQL data cleanup ① Set up a scheduled data deletion task to be cleaned up annually. The data is valid for 3 years and the statistical table of data older than 3 years will be automatically deleted when the deadline expires. ② Manually write a shell script for detection and put it on the server. Check the disk space daily. When the disk space is less than 10%, delete files according to importance and data storage time strategy until the disk space is more than 10%. ③ For MySQL database creation events, check the data volume of a single table daily. If the data volume of a single table exceeds 100 million, write data to the alarm table and notify the platform maintenance personnel to check and handle it.

6. A massive data processing system based on heterogeneous data sources, characterized in that, The processing system includes: The storage module is used to write service call data to storage; the storage includes MongoDB and Elasticsearch; the service call data includes API service call time, apiId, application name, parameters, time consumption, error information, and error code information records in JSON format; writing the service call data to storage includes: S11. Asynchronously write the specific information of the service call data to MongoDB for data statistics queries, and create a new collection for MongoDB every day; S12. Asynchronously write the specific information of the service call data to Elasticsearch for user queries, and create new indexes for Elasticsearch on a rolling basis every month; S13. Every 10 seconds, perform statistics on the number of API calls, average execution time, number of failures, and number of calls for MongoDB data, categorized by year, month, day, and hour, and save the statistical data to a MySQL statistics table; S14. Create new data tables for MySQL every year; The query module is used to query service call data; The cleanup module is used to clean up data files on a regular schedule. The early warning module is used for intelligent early warning via shell scripts, including: S41. Write an intelligent early warning shell script, and then create the xtyj.sh file in / home / opt; S42. Query disk space. If the disk space utilization rate reaches 85%, send an XT_YJ_TOPIC message to Kafka. The message content includes the server IP, CPU and disk usage. S43. Set a scheduled task for the intelligent early warning shell script to execute the script at 1:00 every day; S44. The backend code listens for messages in the corresponding Kafka topic XT_YJ_TOPIC, and after receiving the message, sends an alert message to the system administrator; S45. After receiving the warning message, the system administrator checks for server anomalies and takes appropriate action.

7. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the massive data processing method based on heterogeneous data sources as described in any one of claims 1-5.

Images