Low-frequency data table and cleaning method, system, equipment and product of low-frequency data table operation

Abstract

The invention provides a low-frequency data table and an operation cleaning method, system, equipment and product thereof, and the method comprises the steps: removing a data table used by a front-end billboard from an original data table set to obtain a first low-frequency data table set, and removing a data table used by a front-end billboard from the first low-frequency data table set to obtain a second low-frequency data table set; screening out data tables of which the traffic in the application program development interface is smaller than a first preset value to form a second low-frequency data table set; screening out interface data services of which the calling times are smaller than a second preset value from the original interface data service set to form a low-frequency interface data service set; all data operation tasks corresponding to the data tables in the second low-frequency data table set are screened out from the original data operation task set, and a low-frequency data operation task set is formed; and offline the second low-frequency data table set, and closing a low-frequency interface data service and a data operation task. Storage space waste of the low-frequency data table can be reduced, accumulation of useless codes of the system is reduced, the system operation cost is reduced, and the system operation efficiency is improved.

Description

Background Art

[0002] In modern enterprises, the management of data is very important, and various data can be stored and processed through a data intelligence platform. The data intelligence platform can be divided according to business, including an accommodation data intelligence platform, a flight ticket data intelligence platform, a catering data intelligence platform, etc.

[0003] In the data intelligence platform, there are various data tables based on various cluster types to provide data services. The cluster types include clickhouse, starrocks, MySQL, etc. These data tables provide front-end data display (such as the display of the front-end dashboard) and interface data services (such as Daas interface data service, Data as a service). There are many data operation tasks (such as zeus tasks for data production and synchronization) around these data tables.

[0004] In the existing scenario, in the data intelligence platform, since the logs between different cluster types are not interconnected, the services and applications involved in the data tables are numerous and complex. Low-frequency data tables that have not been used for a long time can only be processed by manual investigation and manual offline. The interface data services that are not in use and the data operation tasks corresponding to the low-frequency data tables also need to be manually verified and closed.

[0005] However, the number of data tables, interface data services, and data operation tasks corresponding to the data tables distributed among different cluster types is huge. Manually verifying and taking them offline one by one is inefficient, resulting in the backlog of low-frequency data and consuming unnecessary storage resources. At the same time, the existence of interface data services with low-frequency usage and data operation tasks corresponding to low-frequency data tables will cause the accumulation of useless code in the system, increase the system operation cost, and reduce the system operation efficiency.

[0006] It should be noted that the information disclosed in the above background art section is only used to strengthen the understanding of the background of the present disclosure, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0007] In view of this, the present disclosure provides a method, system, device, and product for cleaning low-frequency data tables and their operations, so as to at least solve the problem of waste of storage resources caused by the backlog of existing low-frequency data, and the accumulation of useless code in the system caused by interface data services with low-frequency usage and data operation tasks corresponding to low-frequency data tables, increase the system operation cost, and reduce the system operation efficiency.

[0008] On the one hand, an embodiment of the present disclosure provides a method for cleaning a low-frequency data table and its operations, including:

[0009] Acquire a data table used by the front-end dashboard, remove the data table used by the front-end dashboard from the original data table set, obtain a first low-frequency data table set, and filter out data tables whose traffic in the application development interface of the original interface data service set is less than a first preset value from the first low-frequency data table set to form a second low-frequency data table set;

[0010] Filter out interface data services whose call times are less than a second preset value from the original interface data service set to form a low-frequency interface data service set;

[0011] Based on the correspondence between data tables and data operation tasks, all data operation tasks corresponding to data tables in the second low-frequency data table set are screened out from the original data operation task set to form a low-frequency data operation task set;

[0012] The data tables in the second low-frequency data table set are taken offline, the interface data services in the low-frequency interface data service set are closed, and the data operation tasks in the low-frequency data operation task set are closed.

[0013] In some embodiments, obtaining a data table used by the front-end dashboard includes:

[0014] Obtain a first log record of the front-end dashboard; the first log record includes a query statement;

[0015] Get the query statement of the front-end dashboard from the first log record;

[0016] Parse the language of the query statement and obtain the data table used in the language.

[0017] In some embodiments, parsing the language of the query statement to obtain a data table used in the language includes:

[0018] Filter query statements corresponding to invalid queries;

[0019] Query statements with the same aggregation structure but different parameters;

[0020] Regular expressions are used to parse the language of the query statement and obtain the data table used in the language.

[0021] In some embodiments, from the first low-frequency data table set, data tables whose traffic in the application development interface of the original interface data service set is less than a first preset value are screened out to form a second low-frequency data table set, including:

[0022] Get the second log record of the interface data service;

[0023] Query the traffic of each data table in the first low-frequency data table set in the second log record;

[0024] Construct a second low-frequency data table set consisting of all data tables with traffic less than a first preset value.

[0025] In some embodiments, based on the correspondence between data tables and data operation tasks, filter out all data operation tasks corresponding to the data tables in the second low-frequency data table set from the original data operation task set to form a low-frequency data operation task set, including:

[0026] Obtain the task information of all data operation tasks in the original data operation task set; the task information includes data tables.

[0027] Establish a mapping relationship between each data table and the data operation tasks containing the data table in the task information.

[0028] Filter out all data operation tasks corresponding to the data tables in the second low-frequency data table set to form a low-frequency data operation task set.

[0029] In some embodiments, take offline the data tables in the second low-frequency data table set, close the interface data services in the low-frequency interface data service set, and close the data operation tasks in the low-frequency data operation task set. The offline and close operations are both stored in the third log record.

[0030] In some embodiments, the method for cleaning low-frequency data tables and their operations further includes:

[0031] Obtain error messages through patrol inspection, and screen out the error messages caused by incorrect offline through the third log record.

[0032] Based on the error messages, perform recovery operations; the recovery operations include: recovering data tables, recovering data service interfaces, or recovering data operation tasks.

[0033] Send a manual recovery notice based on the failure of performing the recovery operation.

[0034] On the other hand, an embodiment of the present disclosure also provides a system for cleaning low-frequency data tables and their operations, including:

[0035] A low-frequency data table acquisition module, configured to acquire the data tables used by the front-end dashboard, remove the data tables used by the front-end dashboard from the original data table set to obtain a first low-frequency data table set, and filter out the data tables with traffic less than a first preset value in the application programming interfaces of the original interface data service set from the first low-frequency data table set to form a second low-frequency data table set;

[0036] A low-frequency interface data service acquisition module, configured to filter out the interface data services with call times less than a second preset value from the original interface data service set to form a low-frequency interface data service set;

[0037] A low-frequency data operation task acquisition module, configured to screen out all data operation tasks corresponding to the data tables in the second low-frequency data table set from the set of original data operation tasks based on the correspondence between the data tables and the data operation tasks, and form a low-frequency data operation task set;

[0038] A cleaning module, configured to take the data tables in the second low-frequency data table set offline, close the interface data services in the low-frequency interface data service set, and close the data operation tasks in the low-frequency data operation task set.

[0039] In another aspect, an embodiment of the present disclosure further provides a cleaning device for a low-frequency data table and its operations, including:

[0040] A processor;

[0041] A memory storing computer-readable instructions;

[0042] Wherein, the processor is configured to execute the above-mentioned cleaning method for the low-frequency data table and its operations by executing the computer-readable instructions.

[0043] In another aspect, an embodiment of the present disclosure further provides a computer program product, including computer-readable instructions, which when executed by a processor, implement the above-mentioned cleaning method for the low-frequency data table and its operations.

[0044] The cleaning method, system, device, and product for the low-frequency data table and its operations of the present disclosure screen out the low-frequency data table set through the usage conditions in the front-end dashboard and the application programming interface, and take the low-frequency data in it offline, which can reduce the storage space of the data table without affecting the use of the data table; screen out the low-frequency interface data service set according to the call times, and close the low-frequency interface data services in it, which can reduce the accumulation of useless code in the system, reduce the system operation cost, and improve the system operation efficiency; screen out the low-frequency data operation task set corresponding to the low-frequency data table, and close the low-frequency data operation tasks in it, which can reduce the accumulation of useless code in the system, reduce the system operation cost, and improve the system operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0046] Figure 1 is a flowchart of the steps of a cleaning method for a low-frequency data table and its operations provided by an embodiment of the present disclosure;

[0047] Figure 2 is Figure 1 a detailed flowchart of partial steps of step S110 in

[0048] Figure 3 is Figure 2 a schematic flowchart of steps in

[0049] Figure 4 is Figure 2 a flowchart of a specific step of step S113 in

[0050] Figure 5 is Figure 1 a detailed flowchart of another part of steps of step S110 in

[0051] Figure 6 is Figure 1 a flowchart of a specific step of step S130 in

[0052] Figure 7 is a flowchart of steps further included in a cleaning method for a low - frequency data table and its operations provided by an embodiment of the present disclosure;

[0053] Figure 8 is a module structure diagram of a cleaning system for a low - frequency data table and its operations provided by an embodiment of the present disclosure;

[0054] Figure 9 is a schematic structural diagram of a cleaning device for a low - frequency data table and its operations provided by an embodiment of the present disclosure. Detailed Embodiments

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. Identical reference numerals in the figures denote identical or similar structures, and thus their repetitive description will be omitted.

[0056] When specifically describing, terms such as "first", "second" and the like do not denote any order, quantity or importance, but are only used to distinguish different components. In addition, in the description of the present disclosure, the orientation or positional relationship indicated by terms such as "upper", "lower" and the like is based on the orientation or positional relationship shown in the drawings, which is only for convenience of description and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present disclosure.

[0057] It should be noted that, without conflict, the features in the embodiments of the present disclosure and different embodiments can be combined with each other.

[0058] In one aspect, as Figure 1 shown, an embodiment of the present disclosure provides a method for cleaning a low-frequency data table and its operations, including:

[0059] S110. Obtain the data tables used by the front-end dashboard, remove the data tables used by the front-end dashboard from the original data table set to obtain a first low-frequency data table set, and screen out the data tables with traffic less than a first preset value in the application programming interfaces of the original interface data service set from the first low-frequency data table set to form a second low-frequency data table set;

[0060] S120. Screen out the interface data services with the number of calls less than a second preset value from the original interface data service set to form a low-frequency interface data service set;

[0061] S130. Based on the correspondence between the data tables and the data operation tasks, screen out all the data operation tasks corresponding to the data tables in the second low-frequency data table set from the original data operation task set to form a low-frequency data operation task set;

[0062] S140. Take offline the data tables in the second low-frequency data table set, close the interface data services in the low-frequency interface data service set, and close the data operation tasks in the low-frequency data operation task set.

[0063] It should be noted that the above S110 to S140 are only the labels of the steps, which are used for easy reference and to avoid repeated text. Unless otherwise described, the above and subsequent step labels will not limit the execution order of the steps of this method. In other embodiments, the above steps of this method can also be written and implemented in a swapped order, and this is not limited thereto.

[0064] Specifically, the front-end dashboard is a user interface component used to display and visualize data tables. It visually presents the data tables obtained from the backend to users in the form of charts, etc. The interface data services can include: DaaS (Data as a Service) interface data services, and the present disclosure does not limit this. The data operation tasks can include: Zeus tasks, and the present disclosure does not limit this. The Zeus task is an automated task scheduling tool or system used to manage the operation tasks of data tables, and can automatically execute operations such as data insertion, update, and synchronization according to preset rules and schedules.

[0065] The set of original data table collections represents the collection of all data tables used by the front-end dashboard. The set of original interface data services represents the collection of all interface data services for the set of original data tables. The set of original data operation tasks represents the collection of all data operation tasks for the set of original data tables.

[0066] "Low-frequency" in the first low-frequency data table set can be defined as that the front-end dashboard has not accessed a certain data table within a preset time range, or the number of accesses is less than a preset value. The preset time range can be 1 year, 1 month, 1 week, 1 day, etc. The preset value can be set to natural numbers such as 0, 1, 2, 10, etc. When the preset value is set to 0, the "low-frequency" data is also "zero-frequency" data, that is, unused data. Similarly, "low-frequency" in the second low-frequency data table set, the low-frequency interface data service set, and the low-frequency data operation task set can all be defined similarly. The first preset value and the second preset value can refer to the settings of the foregoing preset values and will not be elaborated here.

[0067] In this embodiment, by screening out the low-frequency data table set through the usage in the front-end dashboard and the application programming interface and taking the low-frequency data offline, it is possible to reduce the storage space of the data table without affecting the use of the data table; by screening out the low-frequency interface data service set according to the call times and closing the low-frequency interface data services therein, it is possible to reduce the accumulation of useless code in the system, reduce the system operation cost, and improve the system operation efficiency; by screening out the low-frequency data operation task set corresponding to the low-frequency data table and closing the low-frequency data operation tasks therein, it is possible to reduce the accumulation of useless code in the system, reduce the system operation cost, and improve the system operation efficiency.

[0068] In some embodiments, the method of the above embodiment can be applied to multiple different data intelligent platforms and different cluster types of the same data intelligent platform. The cluster types include ClickHouse, StarRocks, MySQL, etc. The data intelligent platform can be divided according to business, including accommodation data intelligent platform, flight ticket data intelligent platform, and catering data intelligent platform, etc. Correspondingly, the data stored in the data table can be accommodation data, flight ticket data, catering data, etc., and the present disclosure does not limit this. This embodiment can form a general method to parse the log data of data tables of different data intelligent platforms and different cluster types of the same data intelligent platform to clean up low-frequency data tables and their operations, improving the generality of the present disclosure.

[0069] In some embodiments, such as Figure 2 and Figure 3 shown, in step S110, obtaining the data tables used by the front-end dashboard includes:

[0070] S111, obtaining a first log record of the front-end dashboard; the first log record includes a query statement;

[0071] S112, obtaining a query statement of the front-end dashboard from the first log record;

[0072] S113, parsing the language of the query statement to obtain a data table used in the language.

[0073] Specifically, the first log record is buried through Java annotations, that is, Figure 3 The three "@" in the upper left corner represent Java annotations. Java annotations use the AOP (Aspect Orient Programming) mechanism of the Spring framework to record query statements, such as SQL (Structured Query Language) at the point of "querying data from the database". The content of the first log record includes basic information such as the user of this query, query time, queried database cluster, query statement (SQL) language and parameters, query time, etc. Among them, the SQL language and parameters are separated, which is convenient for the aggregation and storage of query statements (SQL). For example, in the SQL "SELECT a, b FROM table_exampleWHERE c=? AND d=?;", when the user enters different c and d, the SQL language is the same, but the parameters are different. The embedding point of the first log record uses Kafka, which is an asynchronous message queue. It is not necessary to write the user's query statement (SQL) to the first log record immediately (synchronously), which can reduce the query time. Therefore, the query statement (SQL) is first written to the Kafka message queue and then asynchronously written to the first log record.

[0074] In some embodiments, Figure 2 , Figure 3 and Figure 4 As shown, step S113 includes:

[0075] S1131, filtering query statements corresponding to invalid queries;

[0076] S1132, query statements with the same aggregation structure but different parameters;

[0077] S1133. Use regular expressions to parse the language of the query statement and obtain a data table used in the language.

[0078] For the above step S1131, specifically, query statements (SQL) generated by invalid queries are filtered out, for example, the number of rows in a table is checked, because such query statements (SQL) do not represent that the table has traffic.

[0079] For the above step S1132, specifically, query statements (SQL) with the same aggregation structure but different parameters are aggregated to compress the storage size. For example, query statements (SQL) with the aggregation parameter being a question mark "?". Among them, every preset time length (for example, 3 hours), the data of the first log record is retrieved from the Kafka message queue and stored in HIVE, and the data of the first log record is retrieved from HIVE every day and stored in ClickHouse.

[0080] For the above step S1133, specifically, for example, the SQL statement "SELECT a,b FROM table_example WHERE c=? AND d=?;" uses the data table "table_example". The purpose of this step is to parse out which data tables are used in an SQL query to finally obtain all the data tables used by the front-end dashboard. The method is regular expressions. After matching "FROM" or "JOIN" without case sensitivity, a continuous string that does not start with a left parenthesis is matched, because the tables used are always after "FROM" or "JOIN", and subqueries always start with a left parenthesis.

[0081] In some embodiments, as Figure 5 shown, from the first set of low-frequency data tables, data tables with traffic less than the first preset value in the application programming interfaces of the original interface data service set are filtered out to form a second set of low-frequency data tables, including:

[0082] S114. Obtain the second log record of the interface data service;

[0083] S115. Query the traffic of each data table in the first set of low-frequency data tables in the second log record;

[0084] S116. Form a second set of low-frequency data tables with all data tables having traffic less than the first preset value.

[0085] Specifically, for the above steps S114 to S116, the purpose is to review whether the data tables in the first set of low-frequency data tables are called (i.e., used) by the interface data service (such as DaaS), so as to rule out the situation of non-low-frequency data tables. The following takes the interface data service as DaaS as an example.

[0086] Check whether a certain data table has usage records in DaaS. The principle is to query the second log record of DaaS. Since the data volume is very large, the second log record of DaaS does not provide the original data. Therefore, instead of parsing the second log record to determine which data tables have traffic, it is to check whether a specific data table has traffic in DaaS. Through the interface of OPS Open API, it is possible to check whether a data table has traffic in the DaaS log within a specified time range. The interface of OPS Open API has three input parameters: the UNIX timestamp of the start time, the UNIX timestamp of the end time, and the data table name. In this way, it is possible to query how many usage frequencies a certain data table has within the start time and the end time.

[0087] The following is the pseudo-code for querying in the second log record of DaaS:

[0088] SELECT COUNT(*)AS usage_cnt FROM table_daas_log WHERE record_time>=start_time AND record_time<=end_time AND content LIKE'%table%';

[0089] During the review process of DaaS, although the step size (end time minus start time) can be directly set, for example, to two months, this will cause the DaaS review query to be slow. Therefore, a method of gradually increasing the step size can be designed for review to improve the DaaS review query efficiency. For example, for a data table, the end time of the first query is now, and the start time is 2 hours before the end time. If the result of the first query is that the data table has no traffic, then the end time of the second query is the start time of the first query, and the start time of the second query is 2 days (48 hours) before the end time. If the result of the second query is that the data table still has no traffic, the step size of the third query will be further increased to 2 weeks (336 hours). The fourth time will increase the step size to 2 months (1440 hours). And so on.

[0090] In some embodiments, as Figure 6 shown, step S130 includes:

[0091] S131. Obtain the task information of all data operation tasks in the original data operation task set; the task information includes data tables;

[0092] S132. Establish a mapping relationship between each data table and the data operation tasks containing the data table in the task information;

[0093] S133. Screen out all data operation tasks corresponding to the data tables in the second low-frequency data table set to form a low-frequency data operation task set.

[0094] Taking the data operation task as a Zeus task as an example:

[0095] For the above step S131, specifically, obtain all Zeus tasks. Obtain all Zeus tasks through the Zeus API, including various information such as the content, configuration, Owner, and title of the Zeus tasks.

[0096] For the above step S132, specifically, associate the Zeus task with the data table. A Zeus task includes several source data tables and several target data tables. The idea for obtaining the source data tables is the same as "parsing the data tables from the first log record", that is, the ones after FROM and JOIN. The idea for obtaining the target data tables is relatively simple and is generally written after -tardb and -tartblnames of the Zeus task. The database name and data table name of the target data table can be parsed through regular expressions. The associated Zeus task information is stored in the data table as the mapping relationship between the Zeus task and the data table for subsequent query use.

[0097] In some embodiments, in S140, when the data table is taken offline, there is a gray-scale change, and the data tables in different clusters are taken offline at intervals of a certain time (such as 48 hours). First, use the rename operation for taking offline, and if it doesn't need to be restored for a period of time, then delete it completely to prevent accidental offline.

[0098] In some embodiments, in S140, both the offline and close operations are stored in the third log record for risk prevention and control.

[0099] In some embodiments, as Figure 7 shown, the cleaning method for low-frequency data tables and their operations further includes:

[0100] S151. Obtain error messages through patrol inspection, and screen out the error messages caused by accidental offline through the third log record;

[0101] S152. Based on the error messages, perform recovery operations; the recovery operations include: recovering the data table, recovering the interface data service, or recovering the data operation task;

[0102] S153. Based on the failure of performing the recovery operation, send a manual recovery notice.

[0103] Taking the interface data service as DaaS and the data operation task as a Zeus task as an example:

[0104] For the above step S151, specifically, error reporting is buried through web page normalization, error messages are obtained through Zeus failed task inspection and Daas interface call failure inspection, error messages caused by missing data tables are classified according to the error content, and it is determined whether it is caused by taking offline based on the data table offline records in the third log record.

[0105] For the above step S152, specifically, after identifying that the taken-offline data table is an incorrect take-offline, the original data table and the backed-up data table are automatically renamed and restored, and DaaS and related Zeus tasks are restored through the API.

[0106] For the above step S153, specifically, if the recovery operation fails, a manual recovery notice is sent for manual intervention and manual recovery. The specific notification method can be email alert.

[0107] On the other hand, as Figure 8 shown, an embodiment of the present disclosure further provides a cleaning system for low-frequency data tables and their operations, including:

[0108] A low-frequency data table acquisition module, configured to acquire the data tables used by the front-end dashboard, remove the data tables used by the front-end dashboard from the original data table set to obtain a first low-frequency data table set, and screen out the data tables with traffic less than a first preset value in the application programming interfaces of the original interface data service set from the first low-frequency data table set to form a second low-frequency data table set;

[0109] A low-frequency interface data service acquisition module, configured to screen out the interface data services with the number of calls less than a second preset value from the original interface data service set to form a low-frequency interface data service set;

[0110] A low-frequency data operation task acquisition module, configured to screen out all data operation tasks corresponding to the data tables in the second low-frequency data table set from the original data operation task set based on the correspondence between the data tables and the data operation tasks to form a low-frequency data operation task set;

[0111] A cleaning module, configured to take offline the data tables in the second low-frequency data table set, close the interface data services in the low-frequency interface data service set, and close the data operation tasks in the low-frequency data operation task set.

[0112] In some embodiments, continue to refer to Figure 8, the cleaning system for the low-frequency data table and its operations further includes: a risk control module for obtaining error messages; the error messages include: error reports of accidental offline of the data table, error reports of accidental closure of the data service interface, and error reports of accidental closure of the data operation task; based on the error messages, perform recovery operations; the recovery operations include: recovering the data table, recovering the data service interface, or recovering the data operation task; based on the failure of performing the recovery operation, send a manual recovery notice.

[0113] For the cleaning system of the low-frequency data table and its operations of the present disclosure, the specific technical solutions and technical effects can refer to the foregoing embodiments of the method for cleaning the low-frequency data table and its operations, and will not be elaborated herein.

[0114] On the other hand, as Figure 9 shown, an embodiment of the present disclosure also provides a cleaning device for a low-frequency data table and its operations, including: a processor; a memory storing computer-readable instructions. Among them, the processor is configured to execute the cleaning method of the low-frequency data table and its operations by executing the computer-readable instructions.

[0115] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method, or a program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "platform" here.

[0116] Next, refer to Figure 9 to describe the electronic device 600 according to this embodiment of the present disclosure. Figure 9 The electronic device 600 shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0117] As Figure 9 shown, the electronic device 600 is presented in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0118] Among them, the storage unit 620 stores computer-readable instructions, and the computer-readable instructions can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present disclosure described in the method part of this specification. For example, the processing unit 610 can execute the steps as Figure 1 shown in

[0119] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203.

[0120] The storage unit 620 may also include a program / utilities 6204 having a set (at least one) of program modules 6205. Such program modules 6205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0121] The bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0122] The electronic device 600 may also communicate with one or more external devices 700 (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or may communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be through an input / output (I / O) interface 650. Moreover, the electronic device 600 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0123] For the cleaning device of the low-frequency data table and its operation of the present disclosure, the specific technical solutions and technical effects can refer to the embodiments of the cleaning method of the low-frequency data table and its operation described above, and will not be elaborated here.

[0124] In another aspect, an embodiment of the present disclosure also provides a computer program product. The computer program product includes computer-readable instructions, and the computer-readable instructions are stored in a computer-readable storage medium. A processor of a computing device may read the computer-readable instructions from the computer-readable storage medium, and the processor executes the computer-readable instructions, so that the computing device executes the cleaning method of the low-frequency data table and its operation described in the above various embodiments.

[0125] The computer-readable instructions included in the computer-readable storage medium can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages - such as Java, C++, etc., and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0126] For the computer program product of the present disclosure, the specific technical solution and technical effect can refer to the foregoing low-frequency data table and the embodiments of the cleaning method thereof, which will not be elaborated herein again.

[0127] In summary, for the low-frequency data table and the cleaning method, system, device, and product thereof provided by the present disclosure, the low-frequency data table set is screened out through the usage in the front-end dashboard and the application programming interface, and the low-frequency data therein is taken offline, which can reduce the storage space of the data table without affecting the use of the data table; the low-frequency interface data service set is screened out according to the call times, and the low-frequency interface data service therein is closed, which can reduce the accumulation of useless code in the system, reduce the system operation cost, and improve the system operation efficiency; by screening out the low-frequency data operation task set corresponding to the low-frequency data table and closing the low-frequency data operation tasks therein, the accumulation of useless code in the system can be reduced, the system operation cost can be reduced, and the system operation efficiency can be improved.

[0128] The above content is a further detailed description of the present disclosure in combination with specific optional implementation manners, and it cannot be determined that the specific implementation of the present disclosure is only limited to these descriptions. For those of ordinary skill in the technical field to which the present disclosure pertains, without departing from the concept of the present disclosure, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present disclosure.

Claims

1. A cleaning method for a low-frequency data table and its operation, characterized in that, Including: Obtain the data tables used by the front-end dashboard, remove the data tables used by the front-end dashboard from the set of original data tables to obtain a first set of low-frequency data tables, and screen out the data tables in the application programming interfaces of the original interface data service set with traffic less than a first preset value from the first set of low-frequency data tables to form a second set of low-frequency data tables; Screen out the interface data services with the number of calls less than a second preset value from the original interface data service set to form a set of low-frequency interface data services; Based on the correspondence between the data tables and the data operation tasks, screen out all the data operation tasks corresponding to the data tables in the second set of low-frequency data tables from the original data operation task set to form a set of low-frequency data operation tasks; Take offline the data tables in the second set of low-frequency data tables, close the interface data services in the set of low-frequency interface data services, and close the data operation tasks in the set of low-frequency data operation tasks.

2. The cleaning method of the low-frequency data table according to claim 1 and its operation, characterized in that, The obtaining of the data tables used by the front-end dashboard includes: Obtain the first log record of the front-end dashboard; the first log record includes query statements; Obtain the query statements of the front-end dashboard from the first log record; Analyze the syntax of the query statements to obtain the data tables used in the syntax.

3. The cleaning method of the low-frequency data table according to claim 2 and its operation, characterized in that, The analyzing of the syntax of the query statements to obtain the data tables used in the syntax includes: Filter the query statements corresponding to invalid queries; Aggregate the query statements with the same structure but different parameters; Use regular expressions to analyze the syntax of the query statements to obtain the data tables used in the syntax.

4. The low-frequency data table according to claim 1 and the cleaning method for its operation, characterized in that, The screening out of the data tables in the first set of low-frequency data tables with traffic less than a first preset value in the application programming interfaces of the original interface data service set to form a second set of low-frequency data tables includes: Obtain the second log record of the interface data service; Query the traffic of each data table in the first set of low-frequency data tables in the second log record; Form a second set of low-frequency data tables with all the data tables having traffic less than the first preset value.

5. The cleaning method of the low-frequency data table and its operation according to claim 1, characterized in that The screening out of all the data operation tasks corresponding to the data tables in the second set of low-frequency data tables from the original data operation task set based on the correspondence between the data tables and the data operation tasks to form a set of low-frequency data operation tasks includes: Obtain the task information of all the data operation tasks in the original data operation task set; the task information includes the data tables; Establish a mapping relationship between each data table and the data operation tasks containing the data table in the task information; Screen out all the data operation tasks corresponding to the data tables in the second set of low-frequency data tables to form a set of low-frequency data operation tasks.

6. The cleaning method of the low-frequency data table and its operation according to claim 1, characterized in that, In the taking offline of the data tables in the second set of low-frequency data tables, closing the interface data services in the set of low-frequency interface data services, and closing the data operation tasks in the set of low-frequency data operation tasks, both the taking offline and closing operations are stored in the third log record.

7. The cleaning method of the low-frequency data table and its operation according to claim 6, characterized in that, Also including: Obtain error messages through patrol inspection, and screen out the error messages caused by accidental offline through the third log record; Based on the error messages, perform a recovery operation; The recovery operation includes: recovering the data table, recovering the interface data service, or recovering the data operation task; Based on the failure of performing the recovery operation, send a manual recovery notice.

8. A cleaning system for a low-frequency data table and its operations, characterized in that, Includes: A low-frequency data table acquisition module, configured to acquire the data tables used by the front-end dashboard, remove the data tables used by the front-end dashboard from the original data table set to obtain a first low-frequency data table set, and screen out the data tables in the application programming interfaces of the original interface data service set whose traffic is less than a first preset value from the first low-frequency data table set to form a second low-frequency data table set; A low-frequency interface data service acquisition module, configured to screen out the interface data services with the number of calls less than a second preset value from the original interface data service set to form a low-frequency interface data service set; A low-frequency data operation task acquisition module, configured to screen out all the data operation tasks corresponding to the data tables in the second low-frequency data table set from the original data operation task set based on the correspondence between the data tables and the data operation tasks to form a low-frequency data operation task set; A cleaning module, configured to take offline the data tables in the second low-frequency data table set, close the interface data services in the low-frequency interface data service set, and close the data operation tasks in the low-frequency data operation task set.

9. A cleaning device for a low-frequency data table and its operation, characterized in that, Includes: A processor; A memory storing computer-readable instructions; Wherein, the processor is configured to execute the cleaning method of the low-frequency data table and its operations according to any one of claims 1 to 7 by executing the computer-readable instructions.

10. A computer program product comprising computer-readable instructions, characterized in that, When the computer-readable instructions are executed by the processor, the cleaning method of the low-frequency data table and its operations according to any one of claims 1 to 7 is implemented.

Images