Data processing method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a data processing method, apparatus, computer equipment, and storage medium. The method comprises the following steps: obtaining a data table to be converted; the data table to be converted includes at least one data group to be converted; associating the data table to be converted with a table template tag and a database tag; obtaining a shared table template corresponding to the data table to be converted and including multiple shared data columns with list headers based on the database tag and the table template tag; performing data classification and analysis on each list header to determine a data filling template corresponding to each shared data column; performing data classification on the table data in each data table to be converted to determine a data category tag corresponding to each table data; and generating a shared database table based on all the table data in each data group to be converted, the data category tags corresponding to the table data, and the data filling template corresponding to each shared data column. The present invention improves the efficiency and accuracy of data storage.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a data processing method, device, computer equipment and storage medium. Background Art

[0002] With the development of science and technology, digital information technology has also gradually developed. The information data under different management agencies are very large. Generally, there is a corresponding database under each management agency, which is used to store data from different departments and organizations to facilitate data sharing between departments and organizations.

[0003] In the existing technology, when different departments and organizations need to store data, they manually organize the data according to the data organization requirements issued by the management agency, and then upload the manually organized data to the database for data sharing. However, this method requires a lot of manpower and material resources, is costly, and manual data organization is slow and has low accuracy. Summary of the Invention

[0004] Embodiments of the present invention provide a data processing method, apparatus, computer device, and storage medium to solve the problem that manual data sorting is inefficient and has low accuracy.

[0005] A data processing method, comprising:

[0006] Acquire a data table to be converted; the data table to be converted includes at least one data group to be converted; the data table to be converted is associated with a table template tag and a database tag;

[0007] According to the database tag and the table template tag, a shared table template corresponding to the data table to be converted is obtained; the shared table template includes a plurality of shared data columns; each of the shared data columns has a list header;

[0008] Performing data classification and analysis on the headers of each list to determine a data filling template corresponding to each shared data column;

[0009] Classifying the tabular data in each of the to-be-converted data tables to determine a data category label corresponding to each of the tabular data;

[0010] A shared database table corresponding to the data table to be converted is generated based on all the table data in each of the data groups to be converted, the data category labels corresponding to the table data one by one, and the data filling template corresponding to each of the shared data columns.

[0011] A data processing device, comprising:

[0012] The data table to be converted acquisition module is used to acquire the data table to be converted; the data table to be converted includes at least one data group to be converted; the data table to be converted is associated with a table template tag and a database tag;

[0013] A shared table template acquisition module is used to acquire a shared table template corresponding to the data table to be converted according to the database tag and the table template tag; the shared table template includes a plurality of shared data columns; each shared data column has a list header;

[0014] A data classification and analysis module, configured to perform data classification and analysis on each of the list headers to determine a data filling template corresponding to each of the shared data columns;

[0015] A data classification module, configured to classify the tabular data in each of the data tables to be converted, so as to determine a data category label corresponding to each of the tabular data;

[0016] The database table generation module is used to generate a shared database table corresponding to the data table to be converted based on all the table data in each of the data groups to be converted, the data category labels corresponding to the table data one by one, and the data filling template corresponding to each of the shared data columns.

[0017] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data processing method when executing the computer program.

[0018] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method is implemented.

[0019] The above-mentioned data processing method, device, computer equipment and storage medium, the data processing method obtains a data table to be converted; the data table to be converted includes at least one data group to be converted; the data table to be converted is associated with a table template tag and a database tag; according to the database tag and the table template tag, a shared table template corresponding to the data table to be converted is obtained; the shared table template includes multiple shared data columns with list headers; data classification and analysis are performed on each of the list headers to determine the data filling template corresponding to each of the shared data columns; data classification is performed on the table data in each of the data tables to be converted to determine the data category tag corresponding to each of the table data; based on all the table data in each of the data groups to be converted, the data category tags corresponding to the table data one by one, and the data filling template corresponding to each of the shared data columns, a shared database table corresponding to the data table to be converted is generated.

[0020] The present invention first determines the shared table template based on the database tags of each data table to be converted, so that the resulting shared database table is aligned with the database, improving the accuracy of data storage. Furthermore, by performing data classification and parsing on the list header and classifying the table data, the shared data column to which each table data belongs can be determined. By filling the template with data from different shared data columns, the table data can be effectively and accurately stored in the corresponding shared data column, thereby ensuring that each shared data column of the generated shared database table has the same cataloging, thereby improving the ease of use of the shared database table. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of an application environment of a data processing method according to an embodiment of the present invention;

[0023] Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention;

[0024] Figure 3 is a principle block diagram of a data processing device in one embodiment of the present invention;

[0025] Figure 4 FIG. 1 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] The data processing method provided by the embodiment of the present invention can be applied as follows: Figure 1 Specifically, the data processing method is applied in a data processing system, which includes Figure 1The client and server shown communicate over a network to address the inefficiency and low accuracy of manual data sorting. The client, also known as the user end, refers to the program that corresponds to the server and provides local services to clients. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0028] In one embodiment, if Figure 2 As shown, a data processing method is provided, which is applied in Figure 1 The server in the example is used as an example, and the steps are as follows:

[0029] S10: Acquire a data table to be converted; the data table to be converted includes at least one data group to be converted; the data table to be converted is associated with a table template tag and a database tag;

[0030] Understandably, the data tables to be converted vary across different application domains. For example, a data table to be converted might be a floating population data table or a company employee record table. A data group to be converted refers to a data group within the same table. For example, in a company employee record table, if the same row contains employee information for the same company employee, then the employee information for different employees in each row constitutes the data group to be converted. The database tag refers to the database type of the platform where the data table to be converted will be stored. For example, this database tag might be DM, Oracle, MySQL, PostgreSQL, etc.

[0031] S20: determining a shared table template corresponding to the data table to be converted according to the database tag and the table template tag; the shared table template includes a plurality of shared data columns; each shared data column has a list header;

[0032] It is understandable that different databases store data in different ways, and different tables correspond to different templates. Therefore, in this embodiment, shared table templates corresponding to different databases and different table templates are pre-designed. After obtaining the data table to be converted, the corresponding shared table template can be quickly determined based on the database tag and table template tag associated with the data table to be converted. In this way, a shared table template that meets the requirements (database requirements and template requirements) can be efficiently queried, eliminating the need to regenerate the corresponding shared table template based on different databases and table templates, thereby speeding up the generation efficiency of shared database tables.

[0033] The shared table template contains multiple shared data columns with list headers. As you can see, the list header is used to indicate the meaning of the shared data column. For example, if the list header is "Name," it indicates that the shared data column should contain the name of each user. Each shared data column has the same data type and corresponds to the list header.

[0034] S30: performing data classification and analysis on the headers of each list to determine a data filling template corresponding to each shared data column;

[0035] It can be understood that data classification analysis is to determine the data type of the shared data column corresponding to each list header, and then the data filling template corresponding to the shared data column can be determined based on the data type. Among them, the essence of the data filling template is the data filling rule of the shared data column. For example, the data filling template can be a symbol that the shared data column can be directly filled with data (that is, no data conversion is required), or a symbol that the shared data column needs to perform rule mapping on the filled data before data filling (for example, converting to "yes" or "no", etc.), or a symbol that the shared data column needs to perform intelligent processing on the filled data before data filling. For example, when a shared data column is a library table name column, such as "staff data table", a ZYSJ-UUID will be generated, where ZYSJ is the pinyin initials conversion of the employee data, and UUID is the unique identifier in each data group to be converted. This can avoid the same name when multiple data groups to be converted are stored, thereby improving data distinction; further, the UUID can be formed by a combination of different numbers, letters, etc., and it does not limit the length of the UUID.

[0036] S40: performing data classification on the table data in each of the data tables to be converted, so as to determine a data category label corresponding to each of the table data;

[0037] Understandably, since there are multiple data groups to be converted in the data table to be converted, and each data group to be converted contains multiple tabular data, it is necessary to classify all tabular data to determine the corresponding shared data column to store each tabular data. The data category label represents the data type of the tabular data, so the shared data column in which the tabular data is stored can be determined by the data category label.

[0038] S50: Generate a shared database table corresponding to the data table to be converted based on all table data in each data group to be converted, data category labels corresponding to the table data one by one, and data filling templates corresponding to each shared data column.

[0039] Specifically, after classifying the table data in each of the data tables to be converted to determine the data category labels corresponding to each of the table data, a shared database table corresponding to the data table to be converted can be generated based on all the table data in each of the data groups to be converted, the data category labels corresponding to the table data, and the data filling template corresponding to each of the shared data columns. At this point, the headers of each list in the generated shared database table are the catalogs of each shared data column, so that the data in the shared database table is normalized and has corresponding catalogs.

[0040] In this embodiment, a shared table template is first determined based on the database tags of each data table to be converted. This ensures that the resulting shared database table is aligned with the database, improving data storage accuracy. Furthermore, by performing data classification and parsing on the table headers and classifying the table data, the shared data columns to which each table data belongs are determined. By populating the template with data from different shared data columns, the table data can be efficiently and accurately stored in the corresponding shared data columns. This ensures that each shared data column in the generated shared database table has the same cataloging, thus improving the ease of use of the shared database table.

[0041] In one embodiment, before step S10, that is, before obtaining the data table to be converted, the following steps are further included:

[0042] Receiving a data monitoring instruction including at least one data group to be monitored; one of the data groups to be monitored includes at least one data to be monitored, and one of the data to be monitored corresponds to a data monitoring rule;

[0043] Among them, the data monitoring instruction can be an instruction sent by the user, or it can be an instruction triggered by the server itself after receiving the data group to be monitored. The data to be monitored can be a single data table in the user's local database, or it can be multiple associated data tables. Data monitoring rules refer to monitoring rules set for different data to be monitored. The data monitoring rules can be null value detection, rule detection (such as ID number, mobile phone number, email address), or regular expression detection, logical detection, etc. In this way, the data group to be monitored can be filtered through data monitoring rules to improve the effectiveness of subsequent data storage, thereby reducing the storage pressure of the database.

[0044] Performing monitoring configuration on the data to be monitored according to the data monitoring rules to obtain monitoring configuration information corresponding to each of the data to be monitored;

[0045] It can be understood that after receiving the data monitoring instruction, the data to be monitored in the data monitoring instruction and the data monitoring rules corresponding to the data to be monitored are obtained, and all the data to be monitored are monitored and configured one by one according to the data monitoring rules. For example, assuming that the data to be monitored contains an ID card number, the corresponding configuration rules are ID card format verification and null value verification, then the ID card number is monitored and configured as follows according to the configuration rules. When the position of the ID card number is monitored, it is determined whether the position of the ID card number is empty. If it is not empty, the ID card number at that position is obtained and the ID card number format is verified, such as the number of ID card digits and whether the meaning represented by each position of the ID card number meets the requirements, such as whether there is falsification in the date of birth (such as the date of birth is in the future, it is falsified); if the position of the ID card number is empty, it is confirmed that the ID card number in the data to be monitored is abnormal.

[0046] It can be understood that in the entire monitoring process of the above example, determining whether the position of the ID card number is empty, and whether the number of digits in the ID card number and the meaning represented by each position of the ID card number meet the requirements is the monitoring configuration information corresponding to the ID card number.

[0047] Monitoring the data to be monitored according to the monitoring configuration information corresponding to each of the data to be monitored, and obtaining data monitoring results of each of the data to be monitored;

[0048] It can be understood that after all the data to be monitored have been monitored and configured, that is, after the monitoring configuration information corresponding to each data to be monitored is generated, a corresponding monitoring execution script is generated based on the monitoring configuration information, so that the monitoring system can identify the monitoring execution script and perform a monitoring process on the data to be monitored based on the corresponding data monitoring rules.

[0049] When the data monitoring results of all the data to be monitored in the same data group to be monitored indicate that the monitoring is qualified, the data group to be monitored is recorded as the data group to be converted, and the data table to be converted is generated based on all the data groups to be converted.

[0050] Specifically, after monitoring the data to be monitored according to the monitoring configuration information corresponding to each of the data to be monitored and obtaining the data monitoring results of each of the data to be monitored, if the data monitoring results of all the data to be monitored in the same data group to be monitored are all indicative of qualified monitoring, the data group to be monitored can be recorded as a data group to be converted, and a data table to be converted can be generated based on all the data groups to be converted. If the data monitoring results of the data to be monitored in the same data group to be monitored are indicative of unqualified monitoring, it indicates that there is an abnormality in the data group to be monitored, and the data group to be monitored can be sent to the user to allow the user to modify the abnormal data in the data group to be monitored, and then perform monitoring configuration on the new data group to be monitored to complete the verification; or the data group to be monitored can be directly eliminated.

[0051] In one embodiment, before receiving the data monitoring instruction including at least one data group to be monitored, the method further includes:

[0052] A service connection is established between each sub-server; one sub-server is associated with one process node.

[0053] Among them, the method of establishing a service connection can be to establish a Kafka cluster between each sub-server, and collect the service data groups of each sub-server through a unified Kafka cluster, so that the service data groups of the sub-servers can be in a unified format. Furthermore, in the process of establishing a service connection between each sub-server, SASL verification can be introduced to improve the security of data transmission and data detection between the Kafka cluster and each sub-server. For example, assuming that the personal information of company employees needs to be collected, the sub-server can be the server corresponding to each subsidiary or branch; assuming that the data table of migrant population needs to be collected, the sub-server can be the server corresponding to each province or city.

[0054] Each of the process nodes is subjected to tracking processing, and the service data group output by each of the sub-servers is received according to the preset data collection interface specification.

[0055] Among them, the data collection interface specification refers to the rules for unifying the service data group collection format when collecting service data groups from each sub-server. For example, the data collection interface specification includes but is not limited to the following field requirements: data collection time, data collection IP address, data name, etc.

[0056] Specifically, after establishing a service connection with each sub-server, each of the process nodes is subjected to tracking processing. For example, visual tracking, no tracking and other methods can be used to obtain data of each process node after detecting that the tracking is triggered, and collect the service data groups output by each sub-server according to the preset data acquisition interface specification. That is, when each sub-server returns the service data group, it needs to be output according to the interface specification, thereby improving the configuration efficiency when subsequently configuring data interception rules.

[0057] Furthermore, since there are many connected sub-servers, each sub-server outputs a large number of service data groups, which may cause a sudden increase in system storage pressure. Therefore, after collecting the service data groups output by each sub-server according to the preset data acquisition interface specifications, a corresponding topic library can be established in the Kafka cluster based on the collected service data groups, and then the same type of service data groups can be stored in the same topic library; the stored topic libraries can also be sent to the big data platform through the cloud server for storage. When the service data group and the corresponding process are needed in the future, the service data group can be obtained by pulling the big data platform.

[0058] Data interception rules are configured to perform data interception on all the service data groups according to the data interception rules.

[0059] Data interception rules are used to monitor the service data groups output by the sub-servers for anomalies. Understandably, after collecting the service data groups output by the sub-servers, these service data groups need to be monitored to prevent errors in the service data groups. This allows for a preliminary round of data screening before subsequent data monitoring using data monitoring rules. Furthermore, different data interception rules can be set based on specific application scenarios. Furthermore, these data interception rules can implement interception processing through Hive hooks and import the intercepted service data groups into a designated directory.

[0060] When the service data group is intercepted, the service data group is recorded as an abnormal data group, and all service data groups except the abnormal data group are recorded as the data to be monitored.

[0061] It is understandable that when a service data group is intercepted, it indicates that the service data group is abnormal, which in turn causes it to be intercepted at any node in the transmission process. Therefore, these abnormal data groups need to be discarded, and all service data groups except the abnormal data groups need to be recorded as data groups to be monitored.

[0062] In one embodiment, one shared data column is associated with one header indexing label; step S50, i.e., generating a shared database table corresponding to the data table to be converted based on all table data in each data group to be converted, the data category labels corresponding to the table data, and the data filling template corresponding to each shared data column, includes:

[0063] The tabular data with the same data category label in each of the data groups to be converted are aggregated into the same column to obtain an aggregated data table; the aggregated data table contains multiple columns of aggregated data; one column of the aggregated data is associated with one data category label;

[0064] Specifically, after classifying the tabular data in each of the data tables to be converted to determine the data category labels corresponding to each of the tabular data, the tabular data with the same data category labels in each of the data groups to be converted are consolidated into the same column, thereby obtaining a consolidated data table corresponding to the data table to be converted. In this case, the consolidated data table contains multiple columns of consolidated data, and all tabular data within a column of consolidated data have the same data category label. For example, the name information in each of the data groups to be converted is consolidated into the same column, which is referred to as the name column.

[0065] Filling the consolidated data having the same data category label as the table header label into the shared data column according to the data filling template corresponding to the shared data column associated with the table header label;

[0066] It can be understood that after the tabular data with the same data category label in each of the data groups to be converted are aggregated into the same column to obtain an aggregated data table, each data category label is compared with the header label, and then the aggregated data with the same data category label as the header label is filled into the shared data column according to the data filling template corresponding to the shared data column associated with the header label.

[0067] After each of the consolidated data is filled into the corresponding shared data column, a data table generation statement corresponding to the shared table template is generated and executed to obtain the shared database table.

[0068] Specifically, after the aggregated data having the same data category label as the header labeling label is filled into the shared data column according to the data filling template corresponding to the shared data column associated with the header labeling label, if all aggregated data are filled into the corresponding shared data column, the data filling is completed, and then a data table generation statement corresponding to the shared table template can be generated and executed, so that a shared database table can be obtained, that is, the shared database table has been filled with the aggregated data.

[0069] In one embodiment, after step S50, that is, after generating the shared database table corresponding to the data table to be converted, the method further includes:

[0070] A first data deletion instruction is received; the first data deletion instruction includes first data to be deleted and a first deletion reason corresponding to the first data to be deleted.

[0071] It is understood that the first data to be deleted refers to any data in the shared database table that the user deletes, and the first deletion reason refers to the reason why the user deletes the first data to be deleted. For example, the first deletion reason may include that the first data to be deleted does not meet the requirements or is expired. The first data deletion instruction can be sent by the user or automatically generated when the user enters the first data to be deleted and the corresponding first deletion reason.

[0072] A first data deletion statement is generated based on the first data to be deleted, so as to trigger a data trigger through the first data deletion statement, thereby deleting the first data to be deleted from the shared database table through the data trigger, and adding the first data to be deleted to the first deletion library table.

[0073] It is understandable that the data trigger is used to respond to the corresponding statement to complete operations such as adding data, deleting data, etc. Specifically, after receiving the first data deletion instruction, a first data deletion statement is generated according to the first data to be deleted. The first data deletion statement is used to indicate that the first data to be deleted needs to be deleted from the shared database table, and then the data trigger is triggered by the first data deletion statement, and then the first data to be deleted is deleted from the shared database table through the data trigger, and the first data to be deleted is added to the first deletion library table. It is understandable that the first data to be deleted can also be added to the first deletion library table through the data trigger at this time. Among them, the first deletion library table is used to store data with the same first deletion reason as the first data to be deleted.

[0074] A first deletion tag is generated according to the first deletion reason, and the first deletion tag is associated with the first deletion library table.

[0075] Specifically, after the first data to be deleted is deleted from the shared database table through the data trigger and the first data to be deleted is added to the first deletion library table, a first deletion label is generated according to the first deletion reason. This step is the process of labeling the first deletion reason, such as extracting keywords in the first deletion reason to determine the first deletion label, and associating the first deletion label with the first deletion library table.

[0076] In one embodiment, after associating the first deletion label with the first deletion library table, the method further includes:

[0077] When the second data deletion instruction is received, a second data deletion statement is generated according to the second deletion data in the second data deletion instruction; the second data deletion instruction also includes a second deletion reason corresponding to the second deletion data.

[0078] It is understood that the second data to be deleted refers to any data in the shared database table that the user deletes after deleting the first data to be deleted. The second deletion reason refers to the reason why the user deleted the second data to be deleted. For example, the second deletion reason may include that the second data to be deleted does not meet the requirements or is expired. The second data deletion instruction can be sent by the user or automatically generated when the user enters the second data to be deleted and the corresponding second deletion reason.

[0079] A data trigger is triggered by the second data deletion statement, so as to delete the second deleted data from the shared database table through the data trigger.

[0080] Specifically, after generating a second data deletion statement based on the second deletion data in the second data deletion instruction, the second data deletion statement indicates that the second data to be deleted needs to be deleted from the shared database table, and then the data trigger is triggered by the second data deletion statement, and then the second data to be deleted is deleted from the shared database table through the data trigger.

[0081] A second deletion label is generated according to the second deletion reason, and the second deletion label is compared with the first deletion label.

[0082] Specifically, after the data trigger is triggered by the second data deletion statement to delete the second deleted data from the shared database table through the data trigger, a second deletion label is generated according to the second deletion reason. This step is the process of labeling the second deletion reason, such as extracting keywords in the second deletion reason to determine the second deletion label, and comparing the second deletion label with the first deletion label.

[0083] When the second deletion label is the same as the first deletion label, the second deletion data is added to the first deletion library table.

[0084] Specifically, after comparing the second deletion tag with the first deletion tag, if the second deletion tag is the same as the first deletion tag, it indicates that the reason for the second deletion is the same as the original reason for the first deletion, and the second deletion data can be added to the first deletion library table. Here, the second deletion data can also be added to the first deletion library table through a data trigger.

[0085] When the second deletion label is different from the first deletion label, the second deletion data is added to a second deletion library table, and the second deletion label is associated with the second deletion library table.

[0086] Specifically, after comparing the second deletion tag with the first deletion tag, if the second deletion tag is different from the first deletion tag, it indicates that the second deletion reason is different from the first deletion reason. Then, the second deletion data is added to the second deletion library table through a data trigger, and the second deletion tag is associated with the second deletion library table. The difference between the first deletion library table and the second deletion library table is that the reasons for deleting the data stored in the tables are different.

[0087] It is understandable that in the subsequent steps, if there is a third data to be deleted and a fourth data to be deleted, the above steps will be performed in the same manner, and a third deletion library table, a fourth deletion library table, etc. may be generated, which will not be described in detail here. In this way, different deletion library tables are established according to different deletion reasons, so that when the shared database table is subsequently analyzed for data, it is possible to more quickly determine whether the data storage in the shared database table is reasonable. For example, if one of the reasons for deleting a library table is an error in the data filling format, if there is a large amount of deleted data in the database table (such as more than 1 / 2 of the data in the shared database table), it can be determined whether a new filling format needs to be re-established for this type of data. In this way, the filling format of the shared database table can be improved, and the filling error rate of the data in the shared database table can be reduced.

[0088] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0089] In one embodiment, a data processing device is provided, which corresponds one-to-one to the data processing method in the above embodiment. Figure 3 As shown, the data processing device includes a data table to be converted acquisition module 10, a shared table template acquisition module 20, a data classification and analysis module 30, a data classification module 40 and a database table generation module 50. The functional modules are described in detail as follows:

[0090] The data table to be converted acquisition module 10 is used to acquire a data table to be converted; the data table to be converted includes at least one data group to be converted; the data table to be converted is associated with a table template tag and a database tag;

[0091] A shared table template acquisition module 20 is configured to acquire a shared table template corresponding to the data table to be converted based on the database tag and the table template tag; the shared table template includes a plurality of shared data columns; each shared data column has a list header;

[0092] A data classification and analysis module 30 is used to perform data classification and analysis on each of the list headers to determine a data filling template corresponding to each of the shared data columns;

[0093] A data classification module 40 is configured to classify the tabular data in each of the data tables to be converted, so as to determine a data category label corresponding to each of the tabular data;

[0094] The database table generation module 50 is used to generate a shared database table corresponding to the data table to be converted based on all the table data in each data group to be converted, the data category labels corresponding to the table data one by one, and the data filling template corresponding to each shared data column.

[0095] Preferably, the data processing device further includes:

[0096] A data monitoring instruction receiving module is configured to receive a data monitoring instruction including at least one data group to be monitored; each of the data groups to be monitored includes at least one data to be monitored, and each of the data to be monitored corresponds to a data monitoring rule;

[0097] A data monitoring configuration module is used to perform monitoring configuration on the data to be monitored according to the data monitoring rules, and obtain monitoring configuration information corresponding to each of the data to be monitored;

[0098] A data monitoring module, configured to monitor the data to be monitored according to the monitoring configuration information corresponding to each of the data to be monitored, and obtain a data monitoring result of each of the data to be monitored;

[0099] The data recording module is used to record the data group to be monitored as the data group to be converted when the data monitoring results of all the data to be monitored in the same data group to be monitored indicate that the monitoring is qualified, and generate the data table to be converted based on all the data groups to be converted.

[0100] For the specific definition of the data processing device, please refer to the definition of the data processing method above and will not be repeated here. Each module in the above-mentioned data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the operations corresponding to each of the above modules.

[0101] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data used by the data processing method in the above embodiment. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0102] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the data processing method in the above embodiment is implemented.

[0103] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method in the above embodiment is implemented.

[0104] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0105] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0106] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Obtaining a data table to be converted; the data table to be converted includes at least one data group to be converted; The data table to be converted is associated with a table template tag and a database tag; According to the database tag and the table template tag, a shared table template corresponding to the data table to be converted is obtained; the shared table template includes a plurality of shared data columns; each shared data column has a list header; the data type under each shared data column is the same and corresponds to the list header; Performing data classification and analysis on the headers of each list to determine a data filling template corresponding to each shared data column; The essence of the data filling template is the data filling rule of the shared data column; Classifying the tabular data in each of the to-be-converted data tables to determine a data category label corresponding to each of the tabular data; Generate a shared database table corresponding to the data table to be converted based on all table data in each of the data groups to be converted, data category labels corresponding to the table data one by one, and data filling templates corresponding to the shared data columns; Associating one of the shared data columns with a table header indexing label; generating a shared database table corresponding to the data table to be converted based on all the table data in each of the data groups to be converted, the data category labels corresponding one-to-one to the table data, and the data filling template corresponding to each of the shared data columns, including: The tabular data with the same data category label in each of the data groups to be converted are aggregated into the same column to obtain an aggregated data table; the aggregated data table contains multiple columns of aggregated data; one column of the aggregated data is associated with one data category label; Filling the consolidated data having the same data category label as the table header label into the shared data column according to the data filling template corresponding to the shared data column associated with the table header label; After each of the consolidated data is filled into the corresponding shared data column, a data table generation statement corresponding to the shared table template is generated and executed to obtain the shared database table.

2. The data processing method according to claim 1, wherein: Before obtaining the data table to be converted, the method further includes: Receiving a data monitoring instruction including at least one data group to be monitored; one of the data groups to be monitored includes at least one data to be monitored, and one of the data to be monitored corresponds to a data monitoring rule; Performing monitoring configuration on the data to be monitored according to the data monitoring rules to obtain monitoring configuration information corresponding to each of the data to be monitored; Monitoring the data to be monitored according to the monitoring configuration information corresponding to each of the data to be monitored, and obtaining data monitoring results of each of the data to be monitored; When the data monitoring results of all the data to be monitored in the same data group to be monitored indicate that the monitoring is qualified, the data group to be monitored is recorded as the data group to be converted, and the data table to be converted is generated based on all the data groups to be converted.

3. The data processing method according to claim 2, wherein: Before receiving the data monitoring instruction including at least one data group to be monitored, the method further includes: Establishing a service connection with each sub-server; one sub-server is associated with one process node; Perform tracking processing on each of the process nodes, and receive the service data group output by each of the sub-servers according to the preset data collection interface specification; configuring a data interception rule to intercept data for all the service data groups according to the data interception rule; When the service data group is intercepted, the service data group is recorded as an abnormal data group, and all service data groups except the abnormal data group are recorded as the data groups to be monitored.

4. The data processing method according to claim 1, wherein: After generating the shared database table corresponding to the data table to be converted, the method further includes: receiving a first data deletion instruction; the first data deletion instruction including first data to be deleted and a first deletion reason corresponding to the first data to be deleted; generating a first data deletion statement according to the first data to be deleted, triggering a data trigger through the first data deletion statement, and further deleting the first data to be deleted from the shared database table through the data trigger, and adding the first data to be deleted to a first deletion library table; A first deletion tag is generated according to the first deletion reason, and the first deletion tag is associated with the first deletion library table.

5. The data processing method according to claim 4, wherein: After associating the first deletion label with the first deletion library table, the method further includes: Upon receiving the second data deletion instruction, generating a second data deletion statement according to the second deletion data in the second data deletion instruction; the second data deletion instruction also includes a second deletion reason corresponding to the second deletion data; triggering a data trigger through the second data deletion statement, so as to delete the second deleted data from the shared database table through the data trigger; generating a second deletion label according to the second deletion reason, and comparing the second deletion label with the first deletion label; When the second deletion label is the same as the first deletion label, the second deletion data is added to the first deletion library table.

6. The data processing method according to claim 5, wherein: After comparing the second deletion label with the first deletion label, the method further includes: When the second deletion label is different from the first deletion label, the second deletion data is added to a second deletion library table, and the second deletion label is associated with the second deletion library table.

7. A data processing device, characterized in that: include: A data table to be converted obtaining module, configured to obtain a data table to be converted; the data table to be converted includes at least one data group to be converted; The data table to be converted is associated with a table template tag and a database tag; a shared table template acquisition module, configured to acquire a shared table template corresponding to the data table to be converted based on the database tag and the table template tag; the shared table template comprising a plurality of shared data columns; each shared data column having a list header; the data type under each shared data column being the same and corresponding to the list header; A data classification and analysis module, configured to perform data classification and analysis on each of the list headers to determine a data filling template corresponding to each of the shared data columns; The essence of the data filling template is the data filling rule of the shared data column; A data classification module, configured to classify the tabular data in each of the data tables to be converted, so as to determine a data category label corresponding to each of the tabular data; a database table generating module, configured to generate a shared database table corresponding to the data table to be converted based on all the table data in each of the data groups to be converted, the data category labels corresponding to the table data one by one, and the data filling template corresponding to each of the shared data columns; Associating one of the shared data columns with a table header indexing label; generating a shared database table corresponding to the data table to be converted based on all the table data in each of the data groups to be converted, the data category labels corresponding one-to-one to the table data, and the data filling template corresponding to each of the shared data columns, including: The tabular data with the same data category label in each of the data groups to be converted are aggregated into the same column to obtain an aggregated data table; the aggregated data table contains multiple columns of aggregated data; one column of the aggregated data is associated with one data category label; Filling the consolidated data having the same data category label as the table header label into the shared data column according to the data filling template corresponding to the shared data column associated with the table header label; After each of the consolidated data is filled into the corresponding shared data column, a data table generation statement corresponding to the shared table template is generated and executed to obtain the shared database table.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data processing method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

Images