Data popularity identification method, device and equipment
By obtaining the log data analysis progress in the database and allocating it to idle nodes for asynchronous analysis, the problem of low database data hotness recognition performance in the prior art is solved, and efficient data hotness recognition is achieved.
Patent Information
- Application Number
- CN202411901560.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-25
AI Technical Summary
The existing database data hot recognition methods have problems such as incompatibility, implementation difficulties and low performance, making it difficult to quickly count the access frequency of the identification table in the database.
By obtaining the server cluster log data analysis progress, allocating unparsed log data to idle nodes, determining the working mode according to the node configuration file to parse the log data asynchronously, and aggregating the parsing results to identify the data popularity.
Asynchronous log data allocation and parsing based on distributed cluster architecture is realized, which improves processing speed, reduces waiting time, and improves resource utilization.
Smart Images
Figure CN119883610A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of software technology, and specifically to a data heat identification method, device and equipment. Background Art
[0002] In a database-based data lake, data heat analysis is an important management strategy that divides data into "hot data", "warm data" and "cold data" based on the access frequency and timeliness of the data. By implementing the separation of hot and cold data, the data lake can significantly improve storage cost-effectiveness, optimize read and write performance, ensure fast response of hot data under high concurrency, and reasonably use low-cost storage media to store cold data.
[0003] Existing database data heat identification methods mainly include database product function methods, API-based methods, and SQL log parsing methods. For MYSQL databases, the database product function method can view the execution frequency of operations such as INSERT, UPDATE, and DELETE by viewing the global status or session status. The API-based method can standardize all applications to access the database through a unified API, automatically perform syntax and lexical parsing during access, and record data access. The SQL log parsing method can use a SQL parser to disassemble the SQL recorded in the database log to obtain data access status.
[0004] Most database products only provide global or session statistics at a certain point in time, or sample statistics. However, the heat analysis of data requires as complete access records of statistical data as possible. This requirement has a great impact on the performance of the database, so the functional methods of database products are difficult to implement. API-based methods usually need to provide standard data access interfaces and need to be compatible with various database products. Due to the complex relationships between enterprise-level projects, the technical risks and manpower investment of modifying and adapting access interfaces are relatively large. In addition, the existing methods based on SQL parsers are mostly single-point centralized architecture designs. Due to the capacity limitations of the server itself, facing the massive database logs in the data lake, it is impossible to complete the extraction and parsing of SQL in a short time; at the same time, due to the different syntax of different database products, there are some compatibility issues.
[0005] Therefore, how to overcome the incompatibility, implementation difficulties and low performance problems in the current database data heat analysis and identification methods, and provide a universal, scalable, high-performance data heat identification solution to quickly statistically identify the access frequency of database tables in the database for data heat analysis and identification is a key issue that needs to be solved urgently. Summary of the invention
[0006] The purpose of the embodiments of this specification is to provide a data heat identification method, device and equipment to overcome the problems in existing data heat identification methods.
[0007] The embodiment of this specification provides a data heat identification method, and the data heat identification method includes:
[0008] Get the first business date of the server cluster log data parsing progress;
[0009] Allocating the unparsed log data for the first business date to idle nodes in the server cluster;
[0010] Determining, according to a configuration file of the idle node, a working mode of the idle node to asynchronously parse the allocated log data;
[0011] The parsing results of the log data are aggregated to identify the popularity of the log data.
[0012] In some embodiments, the data heat identification method further includes:
[0013] Determine whether the URL of the log data under the second business date has been written into the MYSQL configuration status library; the second business date is the date before the current date;
[0014] If the URL of the log data under the second business date has been written into the MYSQL configuration status library, obtain the first business date of the server cluster log data parsing progress;
[0015] If the URL of the log data on the second business date is not written into the MYSQL configuration status library, determine whether the log data on the second business date has been uploaded to the COS object storage;
[0016] If the log data under the second business date has been uploaded to COS object storage, the URL of the log data under the second business date is written into the MYSQL configuration status library;
[0017] Get the first business date of the server cluster log data parsing progress;
[0018] If the log data under the second business date is not uploaded to COS object storage, the first business date of the server cluster log data parsing progress is obtained.
[0019] In some embodiments, the data heat identification method further includes:
[0020] Determine whether the first business date is empty;
[0021] If the first business date is empty, initialize the first business date to the date before the current date;
[0022] Allocating the unparsed log data for the first business date to idle nodes in the server cluster;
[0023] If the first business date is not empty, the unparsed log data under the first business date is distributed to idle nodes in the server cluster.
[0024] In some embodiments, the data heat identification method further includes:
[0025] Obtaining the estimated waiting time of multiple nodes in the server cluster and the load data of the current date and the historical load data;
[0026] Determine the idle nodes in the server cluster using a preset priority scheduling model according to the estimated waiting time of the multiple nodes and the load data on the current date and the historical load data;
[0027] The unparsed log data for the first business date is distributed to the idle node.
[0028] In some embodiments, the data heat identification method further includes:
[0029] The working modes include Common mode, Turbo mode and Crazy mode;
[0030] The Common mode determines the number of processes of the idle node according to the configuration file of the idle node;
[0031] The Turbo mode adaptively sets the number of processes of the idle node according to the CPU and memory of the idle node;
[0032] The Crazy mode sets the number of processes of the idle node according to the maximum usage rate of the CPU and memory of the idle node.
[0033] In some embodiments, the data heat identification method further includes:
[0034] Determining the number of processes of the idle node according to the working mode of the idle node;
[0035] The allocated log data is parsed asynchronously based on the multiple processes of the idle node.
[0036] In some embodiments, the data heat identification method further includes:
[0037] Polling the status of multiple processes of the idle node;
[0038] If the process status is abnormal or timed out, close the process;
[0039] If the status of the process is normal, the log data parsing result of the process is written into the MPP database.
[0040] In some embodiments, the data heat identification method further includes:
[0041] Determine the server cluster to which the different heat log data belongs;
[0042] Repairing missing data in the different heat log data;
[0043] The views corresponding to the different heat log data are mapped into physical data tables.
[0044] The embodiment of this specification also provides a data heat identification device, the device comprising:
[0045] An acquisition module, used to acquire the first business date of the server cluster log data parsing progress;
[0046] A distribution module, used for distributing the unparsed log data under the first business date to idle nodes in the server cluster;
[0047] A determination module, configured to determine the working mode of the idle node according to the configuration file of the idle node to asynchronously parse the allocated log data;
[0048] The aggregation module is used to aggregate the parsing results of the log data to identify the popularity of the log data.
[0049] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-mentioned data heat identification method.
[0050] It can be seen from the technical solution of a data heat identification method provided by the above embodiment of this specification that the embodiment of this specification can obtain the first business date of the server cluster log data parsing progress; allocate the unparsed log data under the first business date to the idle nodes in the server cluster; determine the working mode of the idle node according to the configuration file of the idle node to asynchronously parse the allocated log data; aggregate the parsing results of the log data to identify the heat of the log data. Compared with the existing methods, the log data can be asynchronously allocated and parsed based on the distributed cluster architecture, with strong fault tolerance and scalability. In addition, according to the configuration file of the node in the distributed cluster, the working mode that best suits the node can be selected for asynchronous parsing, which can significantly improve the overall parsing processing speed of the distributed cluster, reduce the waiting time between different processes, and improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below.
[0052] Figure 1 is a flow chart of a data heat identification method provided by an embodiment of this specification;
[0053] Figure 2 It is a schematic diagram of the overall architecture of a data heat identification method provided by an embodiment of this specification;
[0054] Figure 3 It is a logic flow chart of the log parsing task allocation monitoring program provided by the embodiment of this specification;
[0055] Figure 4 It is a logic flow chart of the log parsing, reporting and compression management program provided in the embodiment of this specification;
[0056] Figure 5 It is a logic flow chart of the multidimensional log analysis program provided by the embodiment of this specification;
[0057] Figure 6 It is a schematic diagram of the structure of a data heat identification device provided in an embodiment of this specification;
[0058] Figure 7 It is a schematic diagram of the structural composition of a computer device provided in an embodiment of this specification. DETAILED DESCRIPTION
[0059] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.
[0060] According to the access frequency and importance of data, data can be divided into hot data, warm data and cold data for better management and storage of data. Hot data can refer to data with high access frequency and critical to business and applications. Hot data usually needs to be accessed and processed quickly, so it needs to be stored on high-performance, low-latency devices. Warm data can refer to data with moderate access frequency and of certain importance to business and applications. These data also need to be stored and accessed reliably within a certain period of time, and can be stored on devices with lower cost and larger capacity. Cold data can refer to data with low access frequency and less importance to business and applications. These data need to be stored for a long time and do not need to be accessed and processed frequently, so they can be stored on devices with lower cost and larger capacity.
[0061] See also Figure 1 and Figure 2 . Figure 1 A flowchart of a data heat identification method provided in an embodiment of this specification. Figure 2 The overall architecture diagram of a data heat identification method provided in an embodiment of this specification is as follows. In specific implementation, the data heat identification method includes the following steps:
[0062] S101: Obtain the first business date of the server cluster log data parsing progress.
[0063] In some embodiments, it can be determined whether the URL of the log data under the second business date has been written into the MYSQL configuration status library; the second business date is the date before the current date; if the URL of the log data under the second business date has been written into the MYSQL configuration status library, the first business date of the server cluster log data parsing progress can be obtained.
[0064] By obtaining the first business date of the server cluster log data parsing progress, a foundation is laid for the server cluster to parse the log data under the second business date.
[0065] The URL of the log data can represent a network address or path pointing to the location of the log file. This URL can be a path in the local file system (for example, file: / / / var / log / mysql / error.log) or a URL on a remote server (for example, http: / / example.com / logs / application.log). A data table can be created in the MYSQL configuration state library in advance to store the URL of the log data. This data table can contain fields such as the log data ID (primary key), log data type, log data URL, storage timestamp, etc. You can use an appropriate database connection library (such as Python's mysql-connector-python, Java's JDBC, etc.) to establish a connection with the MYSQL configuration state library.
[0066] See also Figure 3, the server cluster can pre-designate a server node as the execution node of the log parsing task allocation monitoring program. The execution node of the log parsing task allocation monitoring program can define the date before the current date as the second business date, and obtain the URL of the log data under the second business date. The execution node of the log parsing task allocation monitoring program can query whether there is a URL of the log data under the second business date in the data table of the MYSQL configuration status library storing the log data URL. If the URL of the log data under the second business date is found in the data table of the MYSQL configuration status library storing the log data URL, it means that the log data under the second business date has been parsed by the server cluster or is being parsed by the server cluster, and the date of the log data currently being parsed by the server cluster can be obtained as the first business date, and then it can be determined whether the log data under the second business date has been parsed by the server cluster or is being parsed by the server cluster.
[0067] In some embodiments, if the URL of the log data under the second business date is not written into the MYSQL configuration status library, it can be determined whether the log data under the second business date has been uploaded to the COS object storage; if the log data under the second business date has been uploaded to the COS object storage, the URL of the log data under the second business date can be written into the MYSQL configuration status library; the first business date of the server cluster log data parsing progress can be obtained.
[0068] By writing the URL of the log data on the second business date into the MYSQL configuration state library, a foundation is laid for the server cluster to parse the log data on the second business date.
[0069] If the URL of the log data under the second business date is not found in the data table of the MYSQL configuration status library storing the log data URL, it means that the log data under the second business date has not been parsed by the server cluster, and it can be determined whether the log data under the second business date has been uploaded to the COS object storage. The execution node of the log parsing task allocation monitoring program can determine whether the log data under the second business date is ready and uploaded to the COS object storage by checking the local file lock. The local file lock is used in a multi-process or multi-threaded environment to ensure mutually exclusive access to shared resources. The file lock can be a simple file, and its existence or non-existence indicates the state of the lock. For example, the execution node of the log parsing task allocation monitoring program can check whether a preset file exists. If it exists, it means that the local file lock is occupied, that is, the log data under the second business date has been uploaded to the COS object storage; if it does not exist, it means that the local file lock is free, that is, the log data under the second business date has not been uploaded to the COS object storage. If the log data under the second business date has been uploaded to the COS object storage, the URL of the log data under the second business date can be written into the data table of the MYSQL configuration status library storing the log data URL. The execution node of the log parsing task allocation monitoring program may obtain the date of the log data currently being parsed by the server cluster as the first business date, and then the server cluster may parse the log data under the second business date.
[0070] In some embodiments, if the log data under the second business date is not uploaded to the COS object storage, the first business date of the server cluster log data parsing progress can be obtained.
[0071] By obtaining the first business date of the server cluster log data parsing progress when the log data on the second business date is not uploaded to the COS object storage, the log data on the first business date can be parsed, thereby improving the utilization rate of the execution node computing resources.
[0072] If the log data under the second business date has not been uploaded to the COS object storage, it indicates that the log data under the second business date is not ready. The execution node of the log parsing task allocation monitoring program can suspend processing the log data under the second business date, and can obtain the date of the log data currently being parsed by the server cluster as the first business date, and then the server cluster can parse the log data under the first business date.
[0073] S102: Allocate the unparsed log data on the first business date to idle nodes in the server cluster.
[0074] In some embodiments, it can be determined whether the first business date is empty; if the first business date is empty, the first business date can be initialized to the date before the current date; the unparsed log data under the first business date can be allocated to idle nodes in the server cluster; if the first business date is not empty, the unparsed log data under the first business date can be allocated to idle nodes in the server cluster.
[0075] By setting the first business date to the date before the current date when the first business date is empty, the initialization of the first business date is completed, laying a foundation for parsing the log data.
[0076] If the first business date obtained is empty, the execution node of the log parsing task allocation monitoring program can set the first business date to the date before the current date. The execution node of the log parsing task allocation monitoring program can allocate the unparsed log data under the first business date to the idle nodes in the server cluster. If the first business date obtained is not empty, the execution node of the log parsing task allocation monitoring program can allocate the unparsed log data under the first business date to the idle nodes in the server cluster.
[0077] In some embodiments, the estimated waiting time of multiple nodes in the server cluster and the load data and historical load data on the current date can be obtained; based on the estimated waiting time of the multiple nodes and the load data and historical load data on the current date, the idle nodes in the server cluster can be determined using a preset priority scheduling model; and the unparsed log data on the first business date can be allocated to the idle nodes.
[0078] By using a preset priority scheduling model to determine the idle nodes in the server cluster, the expected waiting time of each node as well as the load data of the current date and historical load data can be comprehensively considered to more reasonably determine the most suitable node for log data analysis, thereby promoting load balancing of the server cluster and reasonable allocation of resources.
[0079] The execution node of the log parsing task allocation monitoring program can use performance monitoring tools (such as Prometheus, Grafana, Nagios, etc.) to collect the current performance indicators of each node in the server cluster, such as CPU usage, memory usage, I / O waiting time, etc. Based on these performance indicators, the execution node of the log parsing task allocation monitoring program can calculate the expected waiting time of each node based on the queue length and current processing capacity of each node. The execution node of the log parsing task allocation monitoring program can collect the load data of each node at a specific time point on the day, including CPU load, memory usage, network throughput, etc., and can use a database or log system to store these real-time data. The execution node of the log parsing task allocation monitoring program can obtain the load data of the past few days or weeks from the historical database to understand the load pattern and trend of each node. The execution node of the log parsing task allocation monitoring program can pre-create a priority scheduling model, which can calculate the priority of different nodes based on the expected waiting time, the load data of the day, and the historical load data. For example, a priority scheduling model can be built using machine learning algorithms (such as linear regression, decision trees, random forests, etc.). Specifically, the expected waiting time, the load data of the day and the historical load data of different nodes can be input into the priority scheduling model based on the decision tree, and the priority scheduling model based on the decision tree can output the priority of each node. For another example, it is possible to consider using a weighted sum or scoring system to build a priority scheduling model. Specifically, corresponding weight coefficients can be set for the parameters such as the expected waiting time, the load data of the day and the historical load data, and each parameter can be multiplied by its corresponding weight coefficient, and finally the priority of each node is obtained by adding them up. After the priority of each node is calculated according to the preset priority scheduling model, the nodes can be sorted according to the priority score, and the node with the highest score is used as an idle node. The execution node of the log parsing task allocation monitoring program can use a task queue (such as RabbitMQ, Kafka, etc.) to allocate the unparsed log data under the business date to the idle nodes in the server cluster. Preferably, the execution node of the log parsing task allocation monitoring program can continuously monitor the performance and load of different nodes to ensure that the system can automatically adjust the allocation of unparsed log data when the load is high. If the performance of a node is found to be degraded, the unparsed log data can be dynamically reallocated.
[0080] S103: Determine the working mode of the idle node according to the configuration file of the idle node to asynchronously parse the allocated log data.
[0081] In some embodiments, the working modes include Common mode, Turbo mode and Crazy mode; the Common mode determines the number of processes of the idle node according to the configuration file of the idle node; the Turbo mode adaptively sets the number of processes of the idle node according to the CPU and memory of the idle node; the Crazy mode sets the number of processes of the idle node according to the CPU and memory of the idle node at the maximum utilization rate.
[0082] By providing three different working modes, the business needs and system resource conditions of idle nodes can be weighed to select the working mode, which helps idle nodes to parse log data under different scenarios and requirements.
[0083] The Common mode determines the number of processes of the idle node according to the configuration file of the idle node. Specifically, the idle node can determine its number of processes according to the configuration file of the idle node. The configuration file usually contains a preset, fixed number of processes or some rules for calculating the number of processes, which will not be repeated here. The Common mode is simple and direct. In the Common mode, the number of processes of the idle node is controllable and predictable. The Turbo mode adaptively sets the number of processes according to the CPU and memory usage of the idle node. Specifically, the idle node can monitor the resource usage of the node in real time and dynamically adjust the number of processes to optimize resource utilization. When the resource utilization rate is high, the number of processes is increased to improve efficiency; when the resources are tight, the number of processes is reduced to avoid overload. The Turbo mode can flexibly adapt to different workloads and resource conditions. The Crazy mode sets the number of processes of the idle node according to the maximum utilization rate of the CPU and memory of the idle node. Specifically, the idle node can set the number of processes according to the maximum utilization rate of the CPU and memory of the idle node. The idle node will use all the resources of the node as much as possible to achieve the highest processing efficiency. On the premise of ensuring that the server cluster has sufficient redundancy and fault tolerance, idle nodes can use Crazy mode to improve the efficiency of log data parsing.
[0084] In some embodiments, the number of processes of the idle node may be determined according to the working mode of the idle node; and the allocated log data may be parsed asynchronously based on the multiple processes of the idle node.
[0085] By asynchronously parsing log data through multiple processes of idle nodes, the overall parsing and processing speed of the server cluster can be significantly improved, the waiting time between different processes can be reduced, and resource utilization can be improved.
[0086] See also Figure 4, the server cluster can determine whether the idle node already has a running log parsing, reporting and compression management program. If there is no running running log parsing, reporting and compression management program, the server cluster can determine whether the idle node has been registered in the server cluster and whether the idle node is in an idle state and whether the idle node has completed the assigned task. If the idle node has been registered in the server cluster and the idle node is in an idle state and the idle node has completed the assigned task, the server cluster can update the working state of the idle node to a non-idle state. The server cluster can determine the working mode of the idle node according to the configuration file of the idle node. For example, the server cluster sets the working mode of the idle node to Common mode according to the configuration file of the idle node, and the number of processes is a preset fixed value, or is calculated according to the rules provided in the configuration file. For example, the server cluster sets the working mode of the idle node to Turbo mode according to the configuration file of the idle node, and the idle node dynamically adjusts the number of processes according to the real-time CPU and memory usage to optimize resource utilization. For example, the server cluster sets the working mode of the idle node to Crazy mode according to the idle node's configuration file, and the idle node sets the number of processes to the maximum value that can fully utilize the CPU and memory. After the server cluster determines the number of processes of the idle node, each process of the idle node can parse its assigned log data independently and in parallel without waiting for other processes to complete, thereby realizing asynchronous parsing of log data.
[0087] In some embodiments, the status of multiple processes of the idle node can be polled; if the status of the process is abnormal or timed out, the process can be closed; if the status of the process is normal, the log data parsing result of the process can be written into the MPP database.
[0088] By polling the status of multiple processes on idle nodes, it is possible to ensure that abnormal or timed-out processes are handled in a timely manner, and then the corresponding log data can be reallocated to other processes, speeding up the parsing of log data.
[0089] A data table can be created in advance in the MYSQL configuration status library to store the status of each process of the idle node. This data table can contain fields such as process ID (primary key), process duration, process status, storage timestamp, etc. The status of each process in the idle node can be polled. If the status of the process is abnormal or the duration of the process has timed out, the process can be closed and the data table storing the status of each process of the idle node in the MYSQL configuration status library can be updated. If the status of the process is normal, the result of the process's analysis of the log data can be uploaded to the MPP database, and finally the process can be closed and the data table storing the status of each process of the idle node in the MYSQL configuration status library can be updated. The analysis of log data by each process can be to count the access frequency of log data, the time distribution of log data, and the interaction behavior of users and log data. For example, the access frequency of log data can be to count the number of accesses to each log data item (such as URL, API request, user operation, etc.).
[0090] S104: Aggregate the parsing results of the log data to identify the popularity of the log data.
[0091] In some embodiments, the parsing results of the log data may be aggregated to identify the popularity of the log data.
[0092] By aggregating the parsing results of log data, it is easier to identify trends, patterns, and anomalies contained in the log data based on the popularity of the log data.
[0093] See also Figure 5, the server cluster can run a multidimensional log analysis program to aggregate the log data. Aggregation can be by category, by time window, and by multi-dimensional aggregation. The server cluster can aggregate the log data by specific categories (such as URL, user ID, error type, etc.) to calculate the total number or frequency of each category. The server cluster can aggregate the log data by time window (such as minutes, hours, days, etc.) to analyze the time distribution of data heat. The server cluster can combine multiple dimensions (such as time, category, geographic location, etc.) for aggregation analysis to obtain a more comprehensive data view. Based on the aggregation results, the server cluster can count the number of visits to each category or event within a specified time window, calculate the growth rate of visits to each category or event within a continuous time window, and evaluate the uniqueness of each category or event, such as by calculating the ratio of its frequency of occurrence to the total log volume and combining user behavior data (such as clicks, stay time, conversion rate, etc.) to evaluate the heat of the data. Preferably, the server cluster can sort the log data according to the calculated heat index and use charts (such as bar charts, line charts, heat maps, etc.) to display the data heat so as to intuitively understand the data distribution and trend. Function: This flexibility enables the program to adapt to the log analysis requirements of different systems and provide more accurate and useful analysis results.
[0094] In some embodiments, the server cluster to which the different heat log data belongs may be determined; missing data in the different heat log data may be repaired; and views corresponding to the different heat log data may be mapped to physical data tables.
[0095] The current server cluster can identify and classify log data according to the name of the server cluster to which the log data of different popularity belongs, so as to clarify the metadata cluster to which the popular data (i.e. frequently accessed or important data) belongs. This helps to understand the distribution and popularity of log data in different server clusters, thereby optimizing resource allocation, monitoring performance and identifying potential problems.
[0096] In different hot log data, if there is no table with defined schema (i.e. data structure and type), or some fields are missing necessary information, the server cluster can repair some tables without schema and other default information according to the user's default configuration. By repairing tables without schema and other default information, the integrity and consistency of log data can be ensured, thereby improving the accuracy and reliability of different hot log data analysis.
[0097] The server cluster can map the access to views of different heat log data to the access to the underlying table. A view is a virtual table built based on the data of the table. The server cluster can map the access to views of different heat log data to the access to the underlying table to extract the required data from the physical table. This mapping relationship enables the server cluster to flexibly access and query data without directly understanding the structure and complexity of the underlying physical table, which helps optimize query performance because direct access to the table is usually more efficient than accessing the view.
[0098] As can be seen from the above, the method provided in the embodiment of this specification can obtain the first business date of the server cluster log data parsing progress; allocate the unparsed log data under the first business date to the idle nodes in the server cluster; determine the working mode of the idle nodes according to the configuration files of the idle nodes to asynchronously parse the allocated log data; aggregate the parsing results of the log data to identify the popularity of the log data. Compared with the existing methods, the log data can be asynchronously allocated and parsed based on the distributed cluster architecture, with strong fault tolerance and scalability. In addition, according to the configuration files of the nodes in the distributed cluster, the working mode that best suits the nodes can be selected for asynchronous parsing, which can significantly improve the overall parsing processing speed of the distributed cluster, reduce the waiting time between different processes, and improve resource utilization.
[0099] Based on the above method, this specification also proposes an embodiment of a data heat identification device. Figure 6 As shown, the data heat identification device 600 may specifically include the following modules:
[0100] The acquisition module 601 may be used to acquire the first business date of the server cluster log data parsing progress;
[0101] The allocation module 602 may be used to allocate the unparsed log data under the first business date to the idle nodes in the server cluster;
[0102] The determination module 603 may be used to determine the working mode of the idle node according to the configuration file of the idle node to asynchronously parse the allocated log data;
[0103] The aggregation module 604 may be used to aggregate the parsing results of the log data to identify the popularity of the log data.
[0104] In some embodiments, the above-mentioned acquisition module 601 can be specifically used to determine whether the URL of the log data under the second business date has been written into the MYSQL configuration status library; the second business date is the date before the current date; if the URL of the log data under the second business date has been written into the MYSQL configuration status library, obtain the first business date of the server cluster log data parsing progress; if the URL of the log data under the second business date is not written into the MYSQL configuration status library, determine whether the log data under the second business date has been uploaded to the COS object storage; if the log data under the second business date has been uploaded to the COS object storage, write the URL of the log data under the second business date into the MYSQL configuration status library; obtain the first business date of the server cluster log data parsing progress; if the log data under the second business date has not been uploaded to the COS object storage, obtain the first business date of the server cluster log data parsing progress.
[0105] In some embodiments, the above-mentioned allocation module 602 can be specifically used to determine whether the first business date is empty; if the first business date is empty, initialize the first business date to the date before the current date; allocate the unparsed log data under the first business date to the idle nodes in the server cluster; if the first business date is not empty, allocate the unparsed log data under the first business date to the idle nodes in the server cluster.
[0106] In some embodiments, the above-mentioned allocation module 602 can also be specifically used to obtain the estimated waiting time of multiple nodes in the server cluster and the load data and historical load data on the current date; determine the idle nodes in the server cluster using a preset priority scheduling model based on the estimated waiting time of the multiple nodes and the load data and historical load data on the current date; and allocate the unparsed log data on the first business date to the idle nodes.
[0107] In some embodiments, the determination module 603 may be specifically used to determine the number of processes of the idle node according to the working mode of the idle node; and asynchronously parse the allocated log data based on the multiple processes of the idle node.
[0108] In some embodiments, the above-mentioned determination module 603 can also be specifically used to poll the status of multiple processes of the idle node; if the status of the process is abnormal or timed out, the process is closed; if the status of the process is normal, the log data analysis result of the process is written into the MPP database.
[0109] In some embodiments, the aggregation module 604 can be specifically used to determine the server cluster to which the different heat log data belongs; repair missing data in the different heat log data; and map the views corresponding to the different heat log data into physical data tables.
[0110] It should be noted that the units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0111] As can be seen from the above, based on the data heat identification device provided in the embodiment of this specification, the first business date of the server cluster log data parsing progress can be obtained; the unparsed log data under the first business date is allocated to the idle nodes in the server cluster; according to the configuration file of the idle node, the working mode of the idle node is determined to asynchronously parse the allocated log data; the parsing results of the log data are aggregated to identify the heat of the log data. Compared with the existing methods, the log data can be asynchronously allocated and parsed based on the distributed cluster architecture, with strong fault tolerance and scalability. In addition, according to the configuration file of the node in the distributed cluster, the working mode that best suits the node can be selected for asynchronous parsing, which can significantly improve the overall parsing processing speed of the distributed cluster, reduce the waiting time between different processes, and improve resource utilization.
[0112] The embodiment of the present specification also provides a computer device for a data heat identification method, including a processor and a memory for storing processor executable instructions. When the processor is implemented, it can perform the following steps according to the instructions: obtain the first business date of the server cluster log data parsing progress; allocate the unparsed log data under the first business date to the idle nodes in the server cluster; determine the working mode of the idle nodes according to the configuration file of the idle nodes to asynchronously parse the allocated log data; aggregate the parsing results of the log data to identify the heat of the log data.
[0113] In order to complete the above instructions more accurately, refer to Figure 7 As shown, the embodiment of this specification also provides another specific computer device 700, wherein the computer device 700 includes a network communication port 701, a processor 702 and a memory 703, and the above structures are connected through internal cables so that each structure can perform specific data interaction.
[0114] The processor 702 can be specifically used to obtain the first business date of the server cluster log data parsing progress; allocate the unparsed log data under the first business date to the idle nodes in the server cluster; determine the working mode of the idle nodes according to the configuration file of the idle nodes to asynchronously parse the allocated log data; aggregate the parsing results of the log data to identify the popularity of the log data.
[0115] The memory 703 may be specifically used to store corresponding instruction programs.
[0116] In this embodiment, the network communication port 701 can be a virtual port that is bound to different communication protocols so that different data can be sent or received. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. In addition, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wifi chip; it can also be a Bluetooth chip.
[0117] In this embodiment, the processor 702 may be implemented in any appropriate manner. For example, the processor may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. This specification does not limit this.
[0118] In this embodiment, the memory 703 includes a volatile memory and a non-volatile memory. The memory 703 may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function without a physical form is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.
[0119] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0120] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0123] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A data heat identification method, characterized in that: include: Get the first business date of the server cluster log data parsing progress; Allocating the unparsed log data for the first business date to idle nodes in the server cluster; Determining, according to a configuration file of the idle node, a working mode of the idle node to asynchronously parse the allocated log data; The parsing results of the log data are aggregated to identify the popularity of the log data.
2. The method according to claim 1, characterized in that: The first business date for obtaining the server cluster log data parsing progress includes: Determine whether the URL of the log data under the second business date has been written into the MYSQL configuration status library; the second business date is the date before the current date; If the URL of the log data under the second business date has been written into the MYSQL configuration status library, obtain the first business date of the server cluster log data parsing progress; If the URL of the log data on the second business date is not written into the MYSQL configuration status library, determine whether the log data on the second business date has been uploaded to the COS object storage; If the log data under the second business date has been uploaded to COS object storage, the URL of the log data under the second business date is written into the MYSQL configuration status library; Get the first business date of the server cluster log data parsing progress; If the log data under the second business date is not uploaded to COS object storage, the first business date of the server cluster log data parsing progress is obtained.
3. The method according to claim 1, characterized in that: The method further comprises: Determine whether the first business date is empty; If the first business date is empty, initialize the first business date to the date before the current date; Allocating the unparsed log data for the first business date to idle nodes in the server cluster; If the first business date is not empty, the unparsed log data under the first business date is distributed to idle nodes in the server cluster.
4. The method according to claim 1, characterized in that: The distributing the unparsed log data on the first business date to idle nodes in the server cluster includes: Obtaining the estimated waiting time of multiple nodes in the server cluster and the load data of the current date and the historical load data; Determine the idle nodes in the server cluster using a preset priority scheduling model according to the estimated waiting time of the multiple nodes and the load data on the current date and the historical load data; The unparsed log data for the first business date is distributed to the idle node.
5. The method according to claim 1, characterized in that: The working modes include Common mode, Turbo mode and Crazy mode; the Common mode determines the number of processes of the idle node according to the configuration file of the idle node; The Turbo mode adaptively sets the number of processes of the idle node according to the CPU and memory of the idle node; the Crazy mode sets the number of processes of the idle node according to the CPU and memory of the idle node at the maximum utilization rate.
6. The method according to claim 1, characterized in that: The step of determining the working mode of the idle node according to the configuration file of the idle node to asynchronously parse the allocated log data includes: Determining the number of processes of the idle node according to the working mode of the idle node; The allocated log data is parsed asynchronously based on the multiple processes of the idle node.
7. The method according to claim 6, characterized in that: The asynchronously parsing the allocated log data based on the plurality of processes of the idle node includes: Polling the status of multiple processes of the idle node; If the process status is abnormal or timed out, close the process; If the status of the process is normal, the log data parsing result of the process is written into the MPP database.
8. The method according to claim 1, characterized in that: After aggregating the parsing results of the log data to identify the popularity of the log data, the method further includes: Determine the server cluster to which the different heat log data belongs; Repairing missing data in the different heat log data; The views corresponding to the different heat log data are mapped into physical data tables.
9. A data heat identification device, characterized in that: The device comprises: An acquisition module, used to acquire the first business date of the server cluster log data parsing progress; A distribution module, used for distributing the unparsed log data under the first business date to idle nodes in the server cluster; A determination module, configured to determine the working mode of the idle node according to the configuration file of the idle node to asynchronously parse the allocated log data; The aggregation module is used to aggregate the parsing results of the log data to identify the popularity of the log data.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.