Data processing method and device, equipment, medium and program product

Abstract

The invention provides a data processing method which can be applied to the field of financial science and technology and the field of big data technology. The data processing method comprises the steps of obtaining a database table list of a database, wherein the database table list is generated by performing data processing operation on the database; a pre-stored configuration table is read, a first parameter and a second parameter are configured in the configuration table, the first parameter is used for indicating the to-be-called collection statement, and the second parameter is used for triggering generation of a collection instruction for calling the collection statement; in response to a trigger generation collection instruction, calling a collection statement indicated by a first parameter to traverse the library table list, and collecting statistical information; in response to completion of statistical information collection, updating a second parameter in the configuration table to obtain an updated configuration table; wherein the job flow for collecting the statistical information and the job flow of the processing database are mutually decoupled. The invention further provides a data processing device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a data processing method, device, equipment, medium and program product. Background Art

[0002] In the field of data management and analysis, it is crucial to effectively manage and efficiently query the data in the database. In order to optimize the database query performance, it is usually necessary to collect database statistics, such as the number of rows in the table, the distribution of indexes, etc. This information is used by the database management system to generate more efficient query plans. Therefore, how to collect database statistics more flexibly and efficiently is a topic that needs to be studied in this field. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a data processing method, apparatus, device, medium and program product for improving the flexibility and efficiency of collecting database statistical information.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, including: obtaining a library table list of a database, the library table list being generated by performing data processing operations on the database; reading a pre-stored configuration table, the configuration table being configured with a first parameter and a second parameter, the first parameter being used to indicate a collection statement to be called, and the second parameter being used to trigger the generation of a collection instruction for calling the collection statement; in response to triggering the generation of the collection instruction, calling the collection statement indicated by the first parameter to traverse the library table list and collect statistical information; in response to completion of the collection of the statistical information, updating the second parameter in the configuration table to obtain an updated configuration table; wherein the job flow for collecting the statistical information is decoupled from the job flow for processing the database.

[0005] According to an embodiment of the present disclosure, the method further includes: during the processing operation of the database, collecting statistical information, and the job flow for collecting the statistical information shares at least a part of the data processing resources with the job flow for processing the database.

[0006] According to an embodiment of the present disclosure, during the processing operation of the database, collecting statistical information includes: monitoring the operation time of collecting statistical information, generating an instruction to stop collecting statistical information when the operation time exceeds a preset time, and caching the unfinished statistical information; in response to the database completing the data processing operation, generating a re-collection instruction that calls the collection statement to re-collect the statistical information in the cache.

[0007] According to an embodiment of the present disclosure, the first parameter includes a library name collection parameter and a table name collection parameter, the library name collection parameter is used to indicate the library name collection statement, and the table name collection parameter is used to indicate the table name collection statement; in response to triggering the generation of a collection instruction, the collection statement indicated by the first parameter is called to traverse the library and table list, and collecting statistical information includes: in response to receiving the collection instruction, the library name collection statement and the table name collection statement are called in sequence to traverse the library and table list, and the schema name and table name of the library and table list are respectively obtained to form the name statistical information part in the statistical information.

[0008] According to an embodiment of the present disclosure, the second parameter includes a first time parameter, and the first time parameter is used to trigger the generation of a collection instruction that calls a collection statement; before responding to the triggering of the generation of the collection instruction, it also includes: in response to the time of reading the configuration table satisfying the set condition of the first time parameter, generating a collection instruction.

[0009] According to an embodiment of the present disclosure, the second parameter also includes a second time parameter and a third time parameter, the second time parameter is used to indicate the start collection time of the previous collection cycle, and the third time parameter is used to indicate the period interval between the first time parameter and the second time parameter; in response to the completion of statistical information collection, the second parameter in the configuration table is updated, and the updated configuration table includes: in response to the completion of statistical information collection, the collection end time is recorded in the second time parameter to obtain an updated second time parameter; according to the updated second time parameter and the period interval indicated by the third time parameter, the first time parameter is updated to obtain an updated configuration table.

[0010] According to an embodiment of the present disclosure, it also includes: recording time characteristic data generated in the process of collecting statistical information, the time characteristic data is used to characterize the time consumed in collecting statistical information; in response to the completion of statistical information collection, the time characteristic data and statistical information are organized into a log table, and the log table is used for visual display.

[0011] The second aspect of the present disclosure provides a data processing device, including: an acquisition module, used to acquire a library table list of a database, the library table list is generated by performing data processing operations on the database; a reading module, used to read a pre-stored configuration table, the configuration table is configured with a first parameter and a second parameter, the first parameter is used to indicate a collection statement to be called, and the second parameter is used to trigger the generation of a collection instruction that calls the collection statement; a collection module, used to call the collection statement indicated by the first parameter to traverse the library table list in response to triggering the generation of the collection instruction, and collect statistical information; an update module, used to update the second parameter in the configuration table in response to the completion of the collection of statistical information, and obtain an updated configuration table; wherein the job flow for collecting statistical information is decoupled from the job flow for processing the database.

[0012] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0013] The fourth aspect of the present disclosure further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0014] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0015] Based on the above, by configuring a configuration table with a first parameter and a second parameter, the first parameter indicates the collection statement to be called, and the second parameter triggers the generation of a collection instruction for calling the collection statement. When the collection instruction is triggered, the configuration table can be automatically called to call the collection statement to automatically collect the statistical information of the database, thereby realizing the automatic collection of the statistical information of the database. In addition, the configuration table is automatically updated and can be used for the next collection cycle, thereby improving the efficiency of statistical information collection. The introduction of the configuration table makes the collection of statistical information more flexible. By modifying the parameters in the configuration table, the collection statement and trigger conditions can be easily adjusted to meet different business needs. In addition, by decoupling the operation flow of statistical information collection from the operation flow of database data processing, the collection time can be arranged more flexibly, thereby reducing the impact on the timeliness of data processing. Separating the statistical information collection logic from the data processing logic can reduce the coupling degree of the system and improve the maintainability and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 The application scenario diagram of the data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0018] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0019] Figure 3 A structural block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and

[0020] Figure 4 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0022] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0023] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0024] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0025] It should be noted that the data processing method and device disclosed herein can be used for data processing applications in the financial field, and can also be used for data processing applications in any field other than the financial field. The application field of the data processing method and device disclosed herein is not limited.

[0026] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0027] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0029] In the current data warehouse construction, the Massively Parallel Processing (MPP) distributed database is often used. This database product realizes the separation of metadata, computing, and storage. It can support online analytical processing (OLAP) scenarios with millions of tables, tens of millions of objects, and hundreds of PB of data, providing high-performance storage and computing capabilities for massive relational data for upper-layer components and applications. In the statistical information collection method of MPP database, usually: (1) operation and maintenance personnel manually execute scripts to collect statistical information on tables; (2) post-jobs of scripts are used to collect statistical information on tables in daily batch scheduling. However, manual execution of scripts by operation and maintenance personnel is not only labor-intensive and inefficient, but also unable to collect statistical information on tables in a timely manner, which may affect the efficiency of data processing and access; post-jobs may affect the timeliness of data processing when problems occur, and sometimes they are prone to omissions.

[0030] In view of this, an embodiment of the present disclosure provides a data processing method that can be used to collect statistical information in databases such as MPP, including: obtaining a library table list of the database, the library table list is generated by performing data processing operations on the database; reading a pre-stored configuration table, the configuration table is configured with a first parameter and a second parameter, the first parameter is used to indicate a collection statement to be called, and the second parameter is used to trigger the generation of a collection instruction that calls the collection statement; in response to triggering the generation of the collection instruction, calling the collection statement indicated by the first parameter to traverse the library table list and collect statistical information; in response to completion of the collection of the statistical information, updating the second parameter in the configuration table to obtain an updated configuration table; wherein the job flow for collecting the statistical information is decoupled from the job flow for processing the database.

[0031] Figure 1 The application scenario diagram of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0032] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0033] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0035] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0036] It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] The following will be based on Figure 1 The scene described by Figure 2 The data processing method of the disclosed embodiment is described in detail.

[0039] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0040] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S240, and the data processing method can be executed by a server.

[0041] In operation S210, a library table list of a database is obtained, where the library table list is generated by performing data processing operations on the database.

[0042] The database can be, for example, an MPP database or any other type of database, as long as the database has the need to collect statistical information. In the database, data can be processed by adding, deleting, modifying, and querying data, creating and modifying tables, etc. to generate a list of libraries and tables, that is, a list of databases. For example, the tables generated by the data processing operation can be filtered out according to the table name, table comment, metadata, or other identification information to obtain a list of libraries and tables.

[0043] In operation S220, a pre-stored configuration table is read, wherein a first parameter and a second parameter are configured in the configuration table, wherein the first parameter is used to indicate a collection statement to be called, and the second parameter is used to trigger generation of a collection instruction for calling the collection statement.

[0044] The first parameter and the second parameter can be generated when the configuration table is established and then stored in a specific memory. The first parameter can be, for example, a reference to a specific SQL statement or a set of rules describing how to construct these statements. The collection statements can be, for example, predefined SQL queries that can retrieve the required statistical information from the database. The second parameter can be, for example, a timestamp, an event identifier, or any condition that can trigger the statistical information collection logic. When a specific trigger condition is met, a collection instruction can be generated based on the second parameter, which can call the collection statement specified in the first parameter.

[0045] In operation S230 , in response to triggering the generation of a collection instruction, the collection statement indicated by the first parameter is called to traverse the library table list and collect statistical information.

[0046] For each library table, relevant statistical information can be retrieved according to the logic in the collection statement, such as the number of rows in the query table, the distribution of indexes, the data type and distribution of columns, etc. The collected statistical information can be temporarily stored in an appropriate location, such as a specific table in the database, a file in the file system, or a data structure in memory.

[0047] In operation S240, in response to the completion of the collection of statistical information, the second parameter in the configuration table is updated to obtain an updated configuration table.

[0048] The second parameter may contain a series of complex conditions that need to be reset or updated after the collection task is completed in order to prepare for the next collection task. The updated configuration table can be written back to the storage location to ensure that subsequent collection tasks can be executed based on the latest configuration information.

[0049] Please continue reading Figure 2 , the above operations S210 to S240 can be performed in the workflow of collecting statistical information. The workflow of collecting statistical information is decoupled from the workflow of processing the database. This means that the workflow of collecting statistical information and the workflow of processing the database are independent and do not interfere with each other during execution, and the two can be processed in parallel. In other words, even if the database is being read, written or modified in large quantities, the workflow of collecting statistical information can be performed without interrupting or interfering with these operations. Similarly, the workflow of collecting statistical information will not have a negative impact on the processing operations of the database, and will not affect the efficiency of data processing.

[0050] The above-mentioned embodiment is configured with a configuration table having a first parameter and a second parameter. When the collection instruction is triggered, the collection statement can be called to automatically collect the statistical information of the database, and the configuration table can be automatically updated, which can be used for the next collection cycle, thereby improving the efficiency of statistical information collection. The introduction of the configuration table makes the collection of statistical information more flexible. By modifying the parameters in the configuration table, the collection statement and trigger conditions can be easily adjusted to meet different business needs. In addition, by decoupling the operation flow of statistical information collection from the operation flow of database data processing, the collection time can be arranged more flexibly, thereby reducing the impact on the timeliness of data processing. Separating the statistical information collection logic from the data processing logic can reduce the coupling degree of the system and improve the maintainability and scalability of the system.

[0051] Based on the above embodiment, the data processing method may further include: during the database processing operation, collecting statistical information, and the operation flow for collecting statistical information shares at least a part of data processing resources with the operation flow for processing the database.

[0052] Data processing resources may include, for example, CPU, memory, disk I / O, etc., which can be used to perform database processing operations and collect statistical information. Sharing of these resources can be achieved through resource management at the operating system level, resource scheduler inside the database, etc. This means that when the database is performing processing operations (such as data insertion, update, deletion, etc.), the job flow for collecting statistical information can also use the same resources. Under the premise of decoupling the job flow for collecting statistical information from the job flow for processing the database, resource sharing can improve resource utilization efficiency and reduce the cost of using computing resources.

[0053] In some cases where the amount of data to be processed is large, sharing resources may lead to resource runs. If the competition for resources is too intense, it may cause system performance to degrade. To this end, some resource scheduling strategies can be adopted to reduce the negative impact of resource sharing. In some embodiments, during the processing operation of the database, collecting statistical information may include: monitoring the operation time of collecting statistical information, generating an instruction to stop collecting statistical information when the operation time exceeds a preset time, and caching the unfinished statistical information; in response to the database completing the data processing operation, generating a re-collection instruction that calls the collection statement to re-collect the statistical information in the cache.

[0054] For example, a timer can be set for the job of collecting statistical information to record the start time and duration of the job. At the same time, a reasonable time threshold can be preset as a standard for judging whether the job has timed out. This threshold can be set according to the performance of the database, the load situation and the complexity of statistical information collection, and the present disclosure does not make specific limitations. When the recording time of the timer exceeds the preset threshold, the system can generate an instruction to stop collecting statistical information and cache the currently collected but not yet completed statistical information to a temporary storage area, such as a temporary file in memory or on a disk. Listening to the completion event of the data processing job of the database can be achieved, for example, through a database trigger, log analysis or a third-party monitoring tool. When it is detected that the database processing job is completed, a re-collection instruction that calls the collection statement can be generated based on the cached unfinished information and the recorded status.

[0055] It should be noted that when generating a recollection instruction, you should ensure that there are sufficient resources (such as CPU, memory, disk I / O, etc.) to execute the instruction. If the resources are insufficient, you can continue to wait for a while or trigger the resource scheduling mechanism to release the resources. During the recollection process, you should ensure data consistency with the database.

[0056] Through the above-mentioned re-collection process, the time and resource usage of statistical information collection operations can be effectively monitored and managed. When problems occur, the scheduling job can be stopped at any time, thereby giving priority to ensuring the timeliness of data processing.

[0057] Based on the above embodiment, the first parameter may include a library name collection parameter and a table name collection parameter, the library name collection parameter is used to indicate a library name collection statement, and the table name collection parameter is used to indicate a table name collection statement. In this embodiment, both the library name collection parameter and the table name collection parameter are indicative parameters, which may be a string, a template containing an SQL statement, or a reference to a predefined query.

[0058] In response to triggering the generation of a collection instruction, the collection statement indicated by the first parameter is called to traverse the library table list, and the collection of statistical information includes: in response to receiving the collection instruction, the library name collection statement and the table name collection statement are called in sequence to traverse the library table list, and the schema name and table name of the library table list are respectively obtained to form the name statistical information part in the statistical information.

[0059] For example, you can first call the collection statement indicated by the library name collection parameter in the first parameter. This statement executes and returns a list of names of all databases in the current database. In SQL, a schema is a namespace in a database that can contain a collection of database objects such as tables, views, and indexes. Next, for each database, call the collection statement indicated by the table name collection parameter in the first parameter. This statement can execute and return a list of names of all tables in the current database. After collecting the names of all databases and the table names under each database, this information can be organized into the name statistics part of the statistics, including database name, schema name, table name, etc.

[0060] It should be emphasized that the library name and table name are only the statistical information that usually needs to be collected, but they are not the only information that can be collected. It is understandable that in addition to collecting library names and table names, some other collection parameters can also be set in the configuration table, such as index parameters. In other words, the configuration table can increase or decrease the collection parameters according to needs. In this way, not only can the automatic and accurate collection of specific information in the database be achieved, but also the flexible configuration of the collection parameters can be achieved, and the configuration process is simple and easy to operate.

[0061] Based on the above embodiment, the second parameter may include a first time parameter, and the first time parameter is used to trigger the generation of a collection instruction that calls a collection statement. That is, the second parameter may be a configuration set including multiple sub-parameters, which at least includes the first time parameter. The first time parameter sets the time condition that triggers the generation of the collection instruction, which may be a specific time point or a time interval. The specific form of the first time parameter may depend on the requirements and design of the system.

[0062] In this embodiment, before generating a collection instruction in response to a trigger, the data processing method further includes: generating a collection instruction in response to the time of reading the configuration table satisfying a setting condition of a first time parameter.

[0063] The configuration table can be continuously monitored or periodically read to obtain the latest configuration information. As an example, the setting condition can be that the time of reading the configuration table coincides with the first time parameter. For example, the time of reading the configuration table is recorded as a, and the time corresponding to the first time parameter is recorded as b. When a=b, a collection instruction is generated. This collection instruction will trigger the subsequent collection operation, that is, calling the collection statement indicated by the first parameter to traverse the library table list and collect statistical information. In this way, the collection instruction can be automatically generated, thereby realizing the automatic and timed collection of statistical information.

[0064] Based on the above embodiment, the second parameter also includes a second time parameter and a third time parameter. The second time parameter is used to indicate the start collection time of the previous collection cycle, and the third time parameter is used to indicate the period interval between the first time parameter and the second time parameter.

[0065] In this embodiment, the second parameter not only includes the first time parameter for triggering the collection instruction, but also adds a second time parameter and a third time parameter. The two newly added parameters are used to indicate the start collection time and cycle interval of the previous collection cycle respectively. The second time parameter can be a timestamp or a date and time value, indicating the exact time point when the statistical information collection operation was last performed. The third time parameter sets the time interval between the current collection cycle and the previous collection cycle, for example, it can be a fixed time period or a dynamic interval based on a certain condition or event.

[0066] In this embodiment, in response to the completion of statistical information collection, the second parameter in the configuration table is updated, and the updated configuration table includes: in response to the completion of statistical information collection, the collection end time is recorded in the second time parameter to obtain an updated second time parameter; according to the updated second time parameter and the periodic interval indicated by the third time parameter, the first time parameter is updated to obtain an updated configuration table.

[0067] For example, the collection end time can be recorded as the updated second time parameter, and the updated first time parameter can be obtained based on the addition and subtraction operation of the updated second time parameter and the periodic interval, thereby obtaining an updated configuration table. Since the parameter condition of the time period is added, the configuration table can be used periodically and automatically records the end time of collecting statistical information and the start time of the next collection of statistical information. In this way, automatic timed collection of statistical information can be achieved through the continuously updated configuration table. The time period for collecting statistical information can be adjusted according to actual needs, such as increasing the time interval between two collections, thereby reducing the frequency of collection and reducing the frequent use of computing resources; or shortening the time interval between two collections to improve the timeliness of collection.

[0068] Based on the above embodiments, the data processing method may also include: recording the time characteristic data generated in the process of collecting statistical information, the time characteristic data is used to characterize the time consumed in collecting statistical information; in response to the completion of statistical information collection, the time characteristic data and statistical information are organized into a log table, and the log table is used for visual display.

[0069] Time characteristic data may include, for example, the start time, end time, and duration of the entire collection process. This data can be used to evaluate the efficiency of the collection operation, monitor system performance, and identify potential performance bottlenecks. At each stage of collecting statistical information (such as traversing the library table list, executing the collection statement, storing statistical information, etc.), the corresponding timestamp can be recorded. These timestamps can be used to calculate the time consumption of the entire collection process and generate time characteristic data.

[0070] There can be one or more log tables, which can contain the specific content of statistical information and the corresponding time characteristic data. The generated log table is not only used to store data, but also to support visualization. Visualization tools can be, for example, spreadsheet software, chart libraries, data visualization platforms, etc. By importing these log table data into visualization tools, users (such as system administrators) can intuitively view the collection of statistical information, time consumption trends, and potential performance issues, so as to help users better understand the performance characteristics of the database, discover and solve problems in a timely manner, and thus improve the efficiency and timeliness of statistical information collection.

[0071] In the following, according to the above-mentioned multiple embodiments, configuration tables and log tables that can be used for statistical information collection are listed. The configuration table can be shown in Table 1, and the log table can be shown in Table 2. It should be noted that the contents of Table 1 and Table 2 are only examples and are not used to limit the contents and forms of the configuration table and the log table.

[0072] Table 1

[0073]

[0074] Taking the collection of MPP database tables as an example, in the configuration table shown in Table 1, "Data Date" indicates the data date of filling in the configuration information. The first parameter may include "Database Name", "Table English Name", and "Table Chinese Name". "Database Name" can be used to collect the schema name corresponding to the MPP database table of statistical information. The table name may include "Table English Name" and "Table Chinese Name", and the specific language can be set as needed. "Table English Name" can be used to collect the English name corresponding to the MPP database table of statistical information. "Table Chinese Name" can be used to collect the Chinese name corresponding to the MPP database table of statistical information. The second parameter may include "Next Execution Time", "Last Execution Time" and "Cycle Time". "Next Execution Time" indicates the execution time of the next collection of statistical information for the MPP database table corresponding to this configuration. "Last Execution Time" indicates the execution time of the last collection of statistical information for the MPP database table corresponding to this configuration. "Cycle Time" indicates the frequency of collecting statistical information.

[0075] In this embodiment, the configuration table may also be configured with "whether valid" to check whether the configuration record is valid.

[0076] Table 2

[0077]

[0078] Corresponding to Table 1, Table 2 shows log information for collecting statistical information. The log table may include:

[0079] Data date: indicates the data date when the log is inserted;

[0080] Database name: indicates the schema name corresponding to the MPP database table for collecting statistical information;

[0081] Table English name: the English name of the MPP database table that collects statistical information;

[0082] Table Chinese name: indicates the Chinese name of the MPP database table that collects statistical information;

[0083] Period configuration: indicates the frequency of collecting statistical information;

[0084] Execution start time: indicates the execution start time of collecting statistics information;

[0085] Execution end time: indicates the execution end time of the statistical information collection;

[0086] Execution time: indicates the execution time of collecting statistics information.

[0087] Based on the above data processing method, the present disclosure also provides a data processing device. Figure 3 The device is described in detail.

[0088] Figure 3 The structure block diagram of a data processing device according to an embodiment of the present disclosure is schematically shown.

[0089] like Figure 3 As shown, the data processing device 300 of this embodiment includes an acquisition module 310 , a reading module 320 , a collection module 330 and an update module 340 .

[0090] The acquisition module 310 is used to acquire a library table list of a database, which is generated by performing data processing operations on the database. In one embodiment, the acquisition module 310 can be used to perform the operation S210 described above, which will not be described in detail here.

[0091] The reading module 320 is used to read a pre-stored configuration table, in which a first parameter and a second parameter are configured, the first parameter is used to indicate a collection statement to be called, and the second parameter is used to trigger the generation of a collection instruction for calling the collection statement. In one embodiment, the reading module 320 can be used to perform the operation S220 described above, which will not be repeated here.

[0092] The collection module 330 is used to generate a collection instruction in response to a trigger, call the collection statement indicated by the first parameter to traverse the library table list, and collect statistical information. In one embodiment, the collection module 330 can be used to perform the operation S230 described above, which will not be repeated here.

[0093] The updating module 340 is used to update the second parameter in the configuration table in response to the completion of the collection of the statistical information, so as to obtain an updated configuration table. In one embodiment, the updating module 340 can be used to perform the operation S240 described above, which will not be described in detail here.

[0094] Among them, the workflow of collecting statistical information is decoupled from the workflow of processing the database.

[0095] According to an embodiment of the present disclosure, any multiple modules of the acquisition module 310, the reading module 320, the collection module 330 and the update module 340 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 310, the reading module 320, the collection module 330 and the update module 340 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 310, the reading module 320, the collection module 330 and the update module 340 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0096] Figure 4 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown.

[0097] like Figure 4As shown, the electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 to a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include an onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0098] In RAM 403, various programs and data required for the operation of electronic device 400 are stored. Processor 401, ROM 402 and RAM 403 are connected to each other via bus 404. Processor 401 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 402 and / or RAM 403. It should be noted that the program can also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0099] According to an embodiment of the present disclosure, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to the bus 404. The electronic device 400 may further include one or more of the following components connected to the input / output (I / O) interface 405: an input portion 406 including a keyboard, a mouse, etc.; an output portion 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 408 including a hard disk, etc.; and a communication portion 409 including a network interface card such as a LAN card, a modem, etc. The communication portion 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage portion 408 as needed.

[0100] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0101] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.

[0102] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided by the embodiment of the present disclosure.

[0103] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 401. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0104] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 409, and / or installed from the removable medium 411. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0105] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the processor 401, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0106] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0107] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0109] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, characterized in that: include: Obtaining a library table list of a database, wherein the library table list is generated by performing data processing operations on the database; Reading a pre-stored configuration table, wherein the configuration table is configured with a first parameter and a second parameter, wherein the first parameter is used to indicate a collection statement to be called, and the second parameter is used to trigger generation of a collection instruction for calling the collection statement; In response to triggering the generation of the collection instruction, calling the collection statement indicated by the first parameter to traverse the library table list and collect the statistical information; In response to the completion of collecting the statistical information, updating the second parameter in the configuration table to obtain an updated configuration table; The workflow for collecting the statistical information is decoupled from the workflow for processing the database.

2. The data processing method according to claim 1, characterized in that: Also includes: During the processing operation of the database, the statistical information is collected, and the operation flow for collecting the statistical information shares at least a part of data processing resources with the operation flow for processing the database.

3. The data processing method according to claim 2, characterized in that: During the processing operation in the database, collecting the statistical information includes: Monitoring the operation time of collecting the statistical information, and generating an instruction to stop collecting the statistical information when the operation time exceeds a preset time, and caching the unfinished statistical information; In response to the database completing the data processing operation, a re-collection instruction for calling the collection statement is generated to re-collect the statistical information in the cache.

4. The data processing method according to claim 1, characterized in that: The first parameter includes a library name collection parameter and a table name collection parameter, the library name collection parameter is used to indicate a library name collection statement, and the table name collection parameter is used to indicate a table name collection statement; In response to triggering the generation of the collection instruction, calling the collection statement indicated by the first parameter to traverse the library table list, and collecting the statistical information includes: In response to receiving the collection instruction, the library name collection statement and the table name collection statement are called in sequence to traverse the library table list, and the schema name and table name of the library table list are respectively obtained to form the name statistical information part in the statistical information.

5. The data processing method according to claim 1, characterized in that: The second parameter includes a first time parameter, and the first time parameter is used to trigger the generation of a collection instruction that calls the collection statement; before the generation of the collection instruction in response to the trigger, the step further includes: In response to the time of reading the configuration table satisfying the setting condition of the first time parameter, the collection instruction is generated.

6. The data processing method according to claim 5, characterized in that: The second parameter further includes a second time parameter and a third time parameter, wherein the second time parameter is used to indicate the start collection time of the previous collection cycle, and the third time parameter is used to indicate the period interval between the first time parameter and the second time parameter; In response to the completion of collecting the statistical information, updating the second parameter in the configuration table, and obtaining the updated configuration table includes: In response to the completion of the collection of the statistical information, recording the collection end time into the second time parameter to obtain an updated second time parameter; The first time parameter is updated according to the updated second time parameter and the periodic interval indicated by the third time parameter to obtain an updated configuration table.

7. The data processing method according to claim 1, characterized in that: Also includes: Recording time characteristic data generated during the process of collecting the statistical information, wherein the time characteristic data is used to characterize the time consumed in collecting the statistical information; In response to the completion of the collection of the statistical information, the time characteristic data and the statistical information are organized into a log table, and the log table is used for visual display.

8. A data processing device, characterized in that: include: An acquisition module, used for acquiring a library table list of a database, wherein the library table list is generated by performing data processing operations on the database; A reading module, used for reading a pre-stored configuration table, wherein the configuration table is configured with a first parameter and a second parameter, wherein the first parameter is used for indicating a collection statement to be called, and the second parameter is used for triggering generation of a collection instruction for calling the collection statement; A collection module, configured to generate the collection instruction in response to triggering, call the collection statement indicated by the first parameter to traverse the library table list, and collect the statistical information; An updating module, configured to update the second parameter in the configuration table in response to the completion of collecting the statistical information, so as to obtain an updated configuration table; The workflow for collecting the statistical information is decoupled from the workflow for processing the database.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images