Data processing method and apparatus, storage medium, and electronic device

Abstract

The application provides a data processing method and device, a storage medium and an electronic device. The method comprises the following steps: in response to a data processing instruction, determining each data table corresponding to the data processing instruction; determining a first data table and a second data table having an association relationship with the first data table in each data table; a cluster to which the first data table belongs is different from a cluster to which the second data table belongs; determining a user corresponding to the second data table; granting the user a permission to access the first data table, so that the user accesses the first data table based on the permission. By using the method provided in the application, the second data table having the association relationship with the first data table can be determined, and the user corresponding to the second data table is granted the permission to access, so that the user can directly access the first data table without copying the first data table, thereby reducing data redundancy and saving storage resources.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method and apparatus, storage medium and electronic device. Background Technology

[0002] With the development of computer science and technology, big data processing systems are being used more and more widely in various fields. For example, they can be applied to big data processing in banks, which can greatly improve the efficiency of data processing in banks.

[0003] However, banks have multiple independent businesses, and the big data applications corresponding to each business usually copy data from the source data area (raw data area), the public computing and public access area (processed basic subject data) and various big data application areas to the local big data application for subsequent data processing. This results in a lot of redundancy in the data tables and consumes a lot of storage resources. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a data processing method that can reduce data redundancy and save storage resources.

[0005] The present invention also provides a data processing apparatus to ensure the implementation and application of the above method in practice.

[0006] A data processing method, comprising:

[0007] In response to a data processing instruction, determine the respective data tables corresponding to the data processing instruction;

[0008] A first data table and a second data table that are associated with the first data table are identified in each of the data tables; the cluster to which the first data table belongs is different from the cluster to which the second data table belongs.

[0009] Identify the user corresponding to the second data table;

[0010] Grant the user permission to access the first data table, so that the user can access the first data table based on the permission.

[0011] Optionally, in the above method, determining the second data table that is associated with the first data table in each of the data tables includes:

[0012] Obtain the configuration parameters of the first data table, the configuration parameters including the target database IP address and the target table identifier;

[0013] Based on the target database IP address and target table identifier in the configuration parameters of the first data table, a second data table that is associated with the first data table is determined in each of the data tables;

[0014] Obtain the configuration parameters of the second data table. If a new target database IP address and a new target table identifier exist in the configuration parameters of the second data table, determine a new second data table that is associated with the first data table in each of the data tables based on the new target database IP address and the new target table identifier, until the configuration parameters of the new second data table do not contain a new target database IP address and a new target table identifier.

[0015] Optionally, in the above method, determining the user corresponding to the second data table includes:

[0016] Based on the association between the first data table and the second data table, the table information of the first data table, and the table information of the second data table, construct the data table lineage information;

[0017] The user corresponding to the second data table is determined based on the blood relationship information in the data table.

[0018] Optionally, in the above method, granting the user permission to access the first data table includes:

[0019] A data table view is generated based on the first data table and the second data table; the data table view includes virtual table information of the first data table and virtual table information of the second data table;

[0020] Grant the user permission to access the first data table on the data table view.

[0021] Optionally, after determining the data tables corresponding to the data processing instructions, the above method further includes:

[0022] Obtain basic information for each of the data tables;

[0023] Invalid data tables are identified in each of the data tables based on the basic information of each data table.

[0024] Clean up the invalid data table.

[0025] Optionally, after determining the data tables corresponding to the data processing instructions, the above method further includes:

[0026] Obtain the basic information of each of the aforementioned data tables;

[0027] A baseline list of data tables is generated based on the basic information of each data table, and the execution log information of the database of the cluster to which each data table belongs is obtained;

[0028] The execution frequency of each data table to be processed associated with the database is determined based on the execution log information.

[0029] Cold data tables are identified among the various data tables to be processed based on their execution frequency.

[0030] Match the cold data table with the data table baseline list;

[0031] If the cold data table matches the baseline list of data tables, then the cold data table is archived.

[0032] If the cold data table fails to match the baseline list of data tables, the cold data table is cleaned up.

[0033] A data processing apparatus, comprising:

[0034] The first determining unit is used to respond to a data processing instruction and determine the data tables corresponding to the data processing instruction.

[0035] The second determining unit is used to determine, among the various data tables, a first data table and a second data table that is associated with the first data table; the cluster to which the first data table belongs is different from the cluster to which the second data table belongs.

[0036] The third determining unit is used to determine the user corresponding to the second data table;

[0037] An execution unit is configured to grant the user permission to access the first data table, enabling the user to access the first data table based on the permission.

[0038] Optionally, the second determining unit in the aforementioned apparatus includes:

[0039] The first acquisition subunit is used to acquire the configuration parameters of the first data table, the configuration parameters including the target database IP address and the target table identifier;

[0040] The determination subunit is used to determine, based on the target database IP address and target table identifier in the configuration parameters of the first data table, a second data table that is associated with the first data table in each of the data tables;

[0041] The second acquisition subunit is used to acquire the configuration parameters of the second data table. If there is a new target database IP address and a new target table identifier in the configuration parameters of the second data table, the new second data table that is associated with the first data table is determined in each of the data tables according to the new target database IP address and the new target table identifier, until there is no new target database IP address and a new target table identifier in the configuration parameters of the new second data table.

[0042] A storage medium includes storage instructions, wherein, when the instructions are executed, the device in which the storage medium resides executes the data processing method described above.

[0043] An electronic device includes a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described above in the data processing method.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] This invention provides a data processing method, apparatus, storage medium, and electronic device. The method includes: responding to a data processing instruction, determining various data tables corresponding to the data processing instruction; determining a first data table and a second data table associated with the first data table among the various data tables; the cluster to which the first data table belongs is different from the cluster to which the second data table belongs; determining a user corresponding to the second data table; and granting the user permission to access the first data table, enabling the user to access the first data table based on the permission. By applying the method provided by this invention, a second data table associated with the first data table can be determined, and access permissions can be granted to the user corresponding to the second data table, allowing the user to directly access the first data table without copying the first data table, thereby reducing data redundancy and saving storage resources. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 A flowchart of a data processing method provided by the present invention;

[0048] Figure 2A structural example diagram of a big data processing system provided by the present invention;

[0049] Figure 3 A schematic diagram of a data table access process provided by the present invention;

[0050] Figure 4 A flowchart of a data reading process provided by the present invention;

[0051] Figure 5 A schematic diagram of a data access strategy provided by the present invention;

[0052] Figure 6 A flowchart of a process for cleaning up invalid data tables is provided by the present invention;

[0053] Figure 7 An example diagram illustrating a process for hierarchical storage of hot and cold data provided by this invention;

[0054] Figure 8 A schematic diagram of the structure of a data processing device provided by the present invention;

[0055] Figure 9 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] This invention provides a data processing method that can be applied to electronic devices. The flowchart of the method is shown below. Figure 1 As shown, it specifically includes:

[0059] S101: In response to a data processing instruction, determine the respective data tables corresponding to the data processing instruction.

[0060] In this embodiment, each preset cluster can be scanned to obtain a list of data tables for each cluster, and the data table corresponding to the data processing instruction can be determined based on the list of data tables.

[0061] S102: Determine a first data table and a second data table that are associated with the first data table from among the data tables; the cluster to which the first data table belongs is different from the cluster to which the second data table belongs.

[0062] In this embodiment, the first data table can be any one of the various data tables, and the association relationship can be a replication job configuration relationship, that is, the second data table can be a copy table obtained by copying the first data table.

[0063] S103: Determine the user corresponding to the second data table.

[0064] In this embodiment, the user can be a cluster user of the cluster to which the second data table belongs.

[0065] S104: Grant the user permission to access the first data table, so that the user can access the first data table based on the permission.

[0066] In this embodiment, when a user corresponding to the second data table has a need to access the first data table, the user is granted permission to access the first data table, allowing the user to access the first data table based on the access permission.

[0067] By applying the method provided in this invention, a second data table that is associated with the first data table can be identified, and user access permissions corresponding to the second data table can be granted. This allows users to directly access the first data table without having to copy the contents of the first data table to the second data table, thereby reducing data redundancy and saving storage resources.

[0068] In one embodiment of the present invention, based on the above implementation process, optionally, determining the second data table that is associated with the first data table in each of the data tables includes:

[0069] Obtain the configuration parameters of the first data table, the configuration parameters including the target database IP address and the target table identifier;

[0070] Based on the target database IP address and target table identifier in the configuration parameters of the first data table, a second data table that is associated with the first data table is determined in each of the data tables;

[0071] Obtain the configuration parameters of the second data table. If a new target database identifier and a new target table identifier exist in the configuration parameters of the second data table, determine a new second data table that is associated with the first data table in each of the data tables based on the new target database IP address and the new target table identifier, until the configuration parameters of the new second data table do not contain a new target database IP address and a new target table identifier.

[0072] In this embodiment, the configuration parameters may include the source database IP address, the source table identifier, the target database IP address, and the target table identifier.

[0073] Optionally, the configuration parameters for each data table are used to mark the target table to be copied from the source database to the target database. The source database is the database corresponding to the source database IP address, the source table is the table corresponding to the source table identifier, the target database is the database corresponding to the target database IP address, and the target table is the table corresponding to the target table identifier. That is, the configuration parameters record the relationships between the data tables, i.e., the replication job configuration relationships.

[0074] In one embodiment of the present invention, based on the above implementation process, optionally, determining the user corresponding to the second data table includes:

[0075] Based on the association between the first data table and the second data table, the table information of the first data table, and the table information of the second data table, construct the data table lineage information;

[0076] The user corresponding to the second data table is determined based on the blood relationship information in the data table.

[0077] In this embodiment, the table information of the first data table may be the table name of the first data table and the database cluster to which the first data table belongs, and the table information of the second data table may be the table name of the second data table, the database cluster to which the second data table belongs, and the user accessing the second data table.

[0078] Optionally, the data table lineage information may include the source database table name, i.e., the name of the first data table, and the target database table name, i.e., the name of the second data table; specifically, the data table lineage information is shown in Table 1:

[0079]

[0080] Table 1

[0081] Among them, the users who can access the data are users who have access needs. Tables A1 and A2 are the second data tables of Table A, and Table B1 is the second data table of Table B.

[0082] In one embodiment of the present invention, based on the above implementation process, optionally, granting the user permission to access the first data table includes:

[0083] A data table view is generated based on the first data table and the second data table; the data table view includes virtual table information of the first data table and virtual table information of the second data table;

[0084] Grant the user permission to access the first data table on the data table view.

[0085] In this embodiment of the invention, the data table view is a virtual table derived from the first data table and the second data table. Its content is defined by a query and has the structure of a normal data table, but it does not implement data storage.

[0086] In one embodiment of the present invention, based on the above implementation process, optionally, after determining the respective data tables corresponding to the data processing instruction, the method further includes:

[0087] Obtain basic information for each of the data tables;

[0088] Invalid data tables are identified in each of the data tables based on the basic information of each data table.

[0089] Clean up the invalid data table.

[0090] In this embodiment, the basic information may include the name of the data table, creation time, table size, table user, number of table partitions, table partition type, daily increment, and other information.

[0091] Optionally, it can be determined first whether the table name of each data table contains invalid keywords. If the table name contains invalid keywords, the data table with invalid keywords is sent to the user corresponding to the data table. The user's mark on the data table is received. If the mark is a first type mark, the data table is determined to be a business table. If the mark is a second type mark, the data table is determined to be an invalid data table. The first type mark can also carry the business purpose and retention period of the table.

[0092] In this embodiment, invalid data can be cleaned up at preset time intervals.

[0093] In some embodiments, invalid keywords may include one or more of the following: “_bak”, “_temp”, “_tmp”, “_error”, and “_20211020”.

[0094] In some implementations, it is also possible to determine whether each data table in a preset cluster has expired at preset time intervals, and to clean up the expired data tables.

[0095] In one embodiment of the present invention, based on the above implementation process, optionally, after determining the respective data tables corresponding to the data processing instruction, the method further includes:

[0096] Obtain the basic information of each of the aforementioned data tables;

[0097] A baseline list of data tables is generated based on the basic information of each data table, and the execution log information of the database of the cluster to which each data table belongs is obtained;

[0098] The execution frequency of each data table to be processed associated with the database is determined based on the execution log information.

[0099] Cold data tables are identified among the various data tables to be processed based on their execution frequency.

[0100] Match the cold data table with the data table baseline list;

[0101] If the cold data table matches the baseline list of data tables, then the cold data table is archived.

[0102] If the cold data table fails to match the baseline list of data tables, the cold data table is cleaned up.

[0103] In this embodiment, the basic information of the data table may include the data table name, creation time, table size, table user, number of table partitions, table partition type, daily increment, etc. The database execution log information may be SQL execution log information. The execution log information for each time period can be parsed to determine the data tables to be processed associated with the database and the number of times each data table to be processed is accessed during that time period. The execution frequency of the data table is determined based on the number of accesses during each time period, and data tables with an execution frequency less than a preset execution frequency threshold are identified as cold data tables.

[0104] Optionally, matching the cold data table with the data table baseline list can determine whether the cold data table is within the preset business scope. If the match is successful, it means that the cold data table is within the business scope and can be archived. If the match fails, it means that the cold data table is not within the business scope and can be cleaned up.

[0105] The data processing method provided in this embodiment of the invention can be applied to various fields, such as big data processing systems in the banking sector. Figure 2The diagram shown is a structural example of a big data processing system provided in an embodiment of the present invention. The big data processing system includes a data buffer, a source data area, a public computing area, a public access area, a historical archive area, and multiple application computing areas.

[0106] Specifically, for the data buffer, upstream open systems, branches, and external data download raw data and generate files daily at the end of the day, and then transfer the files to the data buffer.

[0107] The source data area is used to check and load data files from the buffer into the data warehouse daily, and to store the original data in a data format and table structure that closely resembles the source system.

[0108] The public computing area is used to copy source data from the source area to the public computing area for processing and calculation. The data is organized and processed according to dimensions such as customers, products, contracts, employees, channels, and accounts to form basic subject data that is used by the entire bank and then used by downstream application computing areas.

[0109] The public access area is used to organize and form fact table data shared by the head office and branches from the upstream source computing area and public computing area according to the bank's digital operation and information application construction needs, so as to meet the big data analysis needs of all business personnel for fixed reports, self-service reports, ad-hoc queries, etc.

[0110] The application computing area is used to copy data from the upstream source data area, public computing area, and public access area to the local application computing area according to the reporting requirements of various counterpart business departments and regulatory departments. It also organizes and generates model data that needs to be reported to various business departments based on their reporting requirements, in order to further support business development.

[0111] The historical archive area is used to archive expired historical data from various application computing areas, making them available for subsequent business personnel to access, audit, and query.

[0112] In embodiments of the present invention, such as Figure 3 The diagram illustrates a data table access process according to an embodiment of the present invention. First, a list of data tables for each big data cluster is scanned to establish a big data table baseline. Starting from any data table, based on the data table's configuration parameters, the source database IP, source table, target database IP, and target table are extracted to establish a mapping relationship between data tables, such as A-A1, B-B1. Then, a list of data lineage relationships between data tables is formed based on the mapping relationship, and a new data access strategy is formulated. If a data table of a big data application cluster needs to be accessed by other big data application clusters, a data table view is created for that table, and read permissions are granted to cluster users with data access needs. This allows other clusters to directly read the physical data on that cluster. The data reading process is as follows: Figure 4 As shown, the application computing area can directly read data tables in the public computing area or data tables in the source data area, and the public computing area can also directly read data tables in the source data area, avoiding the need to copy data before reading it. To clearly explain the data access strategy, an example is given below:

[0113] See Figure 5 This diagram illustrates a data access strategy provided by an embodiment of the present invention. Cluster A creates a data table view of data table A and grants read permissions to users of clusters B and C, allowing clusters B and C to directly access data table A without needing to copy it. Cluster B creates a data table view of data table B and grants read permissions to users of clusters A and C, allowing clusters A and C to directly access data table B without needing to copy it. Cluster C creates a data table view of data table C and grants read permissions to users of clusters A and B, allowing clusters A and B to directly access data table C without needing to copy it.

[0114] By applying the embodiments of the present invention, the new data access strategy, through view authorization, allows other big data clusters to directly read data table information from the source database after user authorization, avoiding the problem of redundant data replication and storage, while saving network bandwidth consumption caused by data replication, greatly improving data flow efficiency, and saving a lot of storage space in the data center.

[0115] In embodiments of the present invention, such as Figure 6 The diagram shows a flowchart of a process for cleaning up invalid data tables according to an embodiment of the present invention, which specifically includes the following steps:

[0116] Step 1: Scan the data tables in each big data area to obtain basic information such as the name, creation date, table size, table user, number of table partitions, table partition type, and daily increment of each data table.

[0117] Step 2: Define matching rules for data table names. For example, data table names containing keywords such as _bak, _temp, _tmp, _error, _20211020, etc., should be marked as backup tables or error tables.

[0118] Step 3: Send the generated list of data tables to each project team and business unit for confirmation. For data tables with real business uses, supplement the business purpose and retention period of the data tables and mark them as business tables. For backup tables during the application launch process or temporary tables generated during program operation, mark them as invalid tables. Establish a baseline list of overall business data tables.

[0119] Step 4: Based on the business table baseline and invalid data table baseline provided by the project team, clean up the invalid data tables regularly.

[0120] Step 5: Regularly retrieve the production data table list weekly, identify new invalid data, and include it in the cleanup plan; compare it with the established business table list, identify newly added business tables, and continue to provide feedback to the project team and business departments to supplement the business purpose and data retention period, and regularly update the business table baseline list to form a closed loop.

[0121] Step Six: Regularly scan the number of partitions and creation dates of the business data table weekly, and compare them with the data table retention period registered in Step Three to identify whether there is any expired data in the current business table data retention range. Include expired data in the cleanup plan.

[0122] Step 7: Clean up the identified invalid data and repeat the process of cleaning up invalid data tables periodically. Continuously update the business table baseline in a closed loop, and regularly compare and identify invalid data for cleaning. This greatly alleviates the pressure on data center space and storage. At the same time, due to the cleaning of invalid data and expired data, the amount of metadata in the big data cluster and the number of data block scans during data access are greatly reduced, significantly improving data access efficiency.

[0123] In this embodiment of the invention, based on the business usage scenario, some data is accessed frequently and has high requirements for CPU resource configuration. In order to improve access speed, the data storage capacity of a single device cannot be too large, and the scanning efficiency of data blocks is improved. Such frequently accessed hot data is stored on ordinary high-density servers. In some business scenarios, the data access frequency is low or even very low or no one accesses it, and the timeliness requirement is low. Such data consumes a lot of storage resources. If it is stored on ordinary high-density servers, a large number of computing servers are required, occupying a lot of data center space. Therefore, cold data is archived to ultra-high-density hosts, which greatly saves the number of machines, reduces procurement costs, and frees up data center space.

[0124] See Figure 7This is an example diagram illustrating a process for hierarchical storage of hot and cold data provided in an embodiment of the present invention. Specifically, it includes: scanning data tables in each big data zone to obtain basic information such as the name, creation date, table size, table user, number of table partitions, table partition type, and daily increment of each data table; daily capture of SQL execution log information from each big data cluster of the big data application; parsing the SQL execution logs, extracting the data table information associated with the SQL, and counting the tables that have been accessed; calculating the execution frequency of all data tables over the past month, and marking data tables that have never been accessed as cold data; comparing cold data tables with the business baseline tables, creating a list of data tables not within the scope of the business baseline tables, marking them as invalid data to be cleaned up, and marking data tables that match the business tables as cold data to be archived; cleaning up invalid data according to a plan, and simultaneously archiving the marked cold data to low-CPU, ultra-high-density storage devices for temporary business data retrieval needs.

[0125] and Figure 1 Corresponding to the method described above, embodiments of the present invention also provide a data processing apparatus for processing data... Figure 1 The specific implementation of the method, the data processing device provided in this embodiment of the invention can be applied to electronic devices, and its structural schematic diagram is shown below. Figure 8 As shown, it specifically includes:

[0126] The first determining unit 801 is used to respond to a data processing instruction and determine the data tables corresponding to the data processing instruction.

[0127] The second determining unit 802 is used to determine a first data table and a second data table that is associated with the first data table from among the various data tables; the cluster to which the first data table belongs is different from the cluster to which the second data table belongs;

[0128] The third determining unit 803 is used to determine the user corresponding to the second data table;

[0129] The execution unit 804 is used to grant the user permission to access the first data table, so that the user can access the first data table based on the permission.

[0130] In one embodiment of the present invention, based on the above-described solution, optionally, the second determining unit includes:

[0131] The first acquisition subunit is used to acquire the configuration parameters of the first data table, the configuration parameters including the target database IP address and the target table identifier;

[0132] The determination subunit is used to determine, based on the target database IP address and target table identifier in the configuration parameters of the first data table, a second data table that is associated with the first data table in each of the data tables;

[0133] The second acquisition subunit is used to acquire the configuration parameters of the second data table. If there is a new target database IP address and a new target table identifier in the configuration parameters of the second data table, the new second data table that is associated with the first data table is determined in each of the data tables according to the new target database IP address and the new target table identifier, until there is no new target database IP address and a new target table identifier in the configuration parameters of the new second data table.

[0134] In one embodiment of the present invention, based on the above-described solution, optionally, the third determining unit 803 is configured as follows:

[0135] Based on the association between the first data table and the second data table, the table information of the first data table, and the table information of the second data table, construct the data table lineage information;

[0136] The user corresponding to the second data table is determined based on the blood relationship information in the data table.

[0137] In one embodiment of the present invention, based on the above-described solution, optionally, the execution unit 804 is configured as follows:

[0138] A data table view is generated based on the first data table and the second data table; the data table view includes virtual table information of the first data table and virtual table information of the second data table;

[0139] Grant the user permission to access the first data table on the data table view.

[0140] In one embodiment of the present invention, based on the above-described solution, optionally, the data processing device further includes:

[0141] The first acquisition unit is used to acquire basic information of each of the data tables;

[0142] The fourth determining unit is used to determine invalid data tables in each of the data tables based on the basic information of each data table;

[0143] The first cleaning unit is used to clean up the invalid data table.

[0144] In one embodiment of the present invention, based on the above-described solution, optionally, the data processing device further includes:

[0145] The second acquisition unit is used to acquire basic information of each of the data tables;

[0146] The generation unit is used to generate a baseline list of data tables based on the basic information of each data table, and to obtain the execution log information of the database of the cluster to which each data table belongs;

[0147] The fifth determining unit is used to determine the execution frequency of each data table to be processed associated with the database based on the execution log information;

[0148] The sixth determining unit is used to determine the cold data table in each of the data tables to be processed based on the execution frequency of each data table to be processed;

[0149] A matching unit is used to match the cold data table with the data table baseline list;

[0150] An archiving unit is used to archive the cold data table if it matches the baseline list of the data tables.

[0151] The second cleaning unit is used to clean up the cold data table if the cold data table fails to match the baseline list of the data table.

[0152] The specific principles and execution processes of each unit and module in the data processing device disclosed in the above embodiments of the present invention are the same as those of the data processing method disclosed in the above embodiments of the present invention. Please refer to the corresponding parts of the data processing method provided in the above embodiments of the present invention, and they will not be repeated here.

[0153] This invention also provides a storage medium, which includes stored instructions, wherein the execution of the instructions controls the device where the storage medium is located to perform the above-described data processing method.

[0154] This invention also provides an electronic device, the structural schematic of which is shown below. Figure 9 As shown, it specifically includes a memory 901 and one or more instructions 902, wherein one or more instructions 902 are stored in the memory 901 and configured to be executed by one or more processors 903 to perform the following operations:

[0155] In response to a data processing instruction, determine the respective data tables corresponding to the data processing instruction;

[0156] A first data table and a second data table that are associated with the first data table are identified in each of the data tables; the cluster to which the first data table belongs is different from the cluster to which the second data table belongs.

[0157] Identify the user corresponding to the second data table;

[0158] Grant the user permission to access the first data table, so that the user can access the first data table based on the permission.

[0159] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0160] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0161] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this invention, the functions of each unit can be implemented in one or more software and / or hardware components.

[0162] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0163] The data processing method provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method, characterized in that, include: In response to a data processing instruction, determine the respective data tables corresponding to the data processing instruction; A first data table and a second data table that are associated with the first data table are identified in each of the data tables; the cluster to which the first data table belongs is different from the cluster to which the second data table belongs. Identify the user corresponding to the second data table; Grant the user permission to access the first data table, so that the user can access the first data table based on the permission; The step of determining a second data table that is associated with the first data table in each of the data tables includes: Obtain the configuration parameters of the first data table, the configuration parameters including the target database IP address and the target table identifier; Based on the target database IP address and target table identifier in the configuration parameters of the first data table, a second data table that is associated with the first data table is determined in each of the data tables; Obtain the configuration parameters of the second data table. If a new target database IP address and a new target table identifier exist in the configuration parameters of the second data table, determine a new second data table that is associated with the first data table in each of the data tables based on the new target database IP address and the new target table identifier, until the configuration parameters of the new second data table do not contain a new target database IP address and a new target table identifier.

2. The method according to claim 1, characterized in that, The step of determining the user corresponding to the second data table includes: Based on the association between the first data table and the second data table, the table information of the first data table, and the table information of the second data table, construct the data table lineage information; The user corresponding to the second data table is determined based on the blood relationship information in the data table.

3. The method according to claim 1, characterized in that, Granting the user access to the first data table includes: A data table view is generated based on the first data table and the second data table; the data table view includes virtual table information of the first data table and virtual table information of the second data table; Grant the user permission to access the first data table on the data table view.

4. The method according to claim 1, characterized in that, After determining the respective data tables corresponding to the data processing instructions, the method further includes: Obtain basic information for each of the data tables; Invalid data tables are identified in each of the data tables based on the basic information of each data table. Clean up the invalid data table.

5. The method according to claim 1, characterized in that, After determining the respective data tables corresponding to the data processing instructions, the method further includes: Obtain basic information for each of the aforementioned data tables; A baseline list of data tables is generated based on the basic information of each data table, and the execution log information of the database of the cluster to which each data table belongs is obtained; The execution frequency of each data table to be processed associated with the database is determined based on the execution log information. Cold data tables are identified among the various data tables to be processed based on their execution frequency. Match the cold data table with the data table baseline list; If the cold data table matches the baseline list of data tables, then the cold data table is archived. If the cold data table fails to match the baseline list of data tables, the cold data table is cleaned up.

6. A data processing apparatus, characterized in that, include: The first determining unit is used to respond to a data processing instruction and determine the data tables corresponding to the data processing instruction. The second determining unit is used to determine, among the various data tables, a first data table and a second data table that is associated with the first data table; The cluster to which the first data table belongs is different from the cluster to which the second data table belongs; The third determining unit is used to determine the user corresponding to the second data table; An execution unit is configured to grant the user permission to access the first data table, enabling the user to access the first data table based on the permission. The second determining unit includes: The first acquisition subunit is used to acquire the configuration parameters of the first data table, the configuration parameters including the target database IP address and the target table identifier; The determination subunit is used to determine, based on the target database IP address and target table identifier in the configuration parameters of the first data table, a second data table that is associated with the first data table in each of the data tables; The second acquisition subunit is used to acquire the configuration parameters of the second data table. If there is a new target database IP address and a new target table identifier in the configuration parameters of the second data table, the new second data table that is associated with the first data table is determined in each of the data tables according to the new target database IP address and the new target table identifier, until there is no new target database IP address and a new target table identifier in the configuration parameters of the new second data table.

7. A storage medium, characterized in that, The storage medium includes storage instructions, wherein when the instructions are executed, the device containing the storage medium is controlled to perform the data processing method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes a memory, and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any one of claims 1 to 5.

Images