A data processing method, device, equipment and medium

Abstract

The embodiment of the present invention provides a data processing method, device, equipment and medium, which are used to solve the problem in the prior art that the data of the same attribute stored in the data table has different column names or field types, so that the data of the same attribute stored in different data tables cannot be effectively merged. In the embodiment of the present invention, for each column in the first data table, the keyword, the first standard name and the standard field type corresponding to the first column name of the column are obtained, and the column name of the column corresponding to the column is recorded as the keyword and the first standard name in the second data table corresponding to the first data table. Therefore, the name of each column in the second data table corresponding to the first data table can be standardized, and the field type of the column corresponding to the column in the second data table can be set to the standard field type, thereby realizing the standardization of the field type of each column in the second data table corresponding to the first data table.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a data processing method, device, equipment and medium. Background Art

[0002] With the development of society, users usually store a large amount of data in the form of data tables. When storing data in a data table, users usually store data of a certain attribute in the same column of the data table. For example, users store the names of different users in the same column, or store the ID numbers of different users in the same column.

[0003] In the prior art, due to the different personal habits of different users, the column names of data with the same attribute stored in the data table are usually different, and the field types of data with the same attribute stored in the data table are different, making it impossible to effectively merge the data with the same attribute stored in different data tables. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method, apparatus, device and medium to solve the problem in the prior art that data with the same attribute stored in a data table has different column names or field types, making it impossible to effectively merge data with the same attribute stored in different data tables.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising:

[0006] For each column of the first data table to be adjusted, the target number and keyword corresponding to the first column name are obtained through the pre-trained model and the first column name of the column; if the target number is included in the correspondence between the pre-saved number and the standard data, the first standard data corresponding to the target number is obtained; a second data table corresponding to the first data table is established, the first standard name and standard field type included in the first standard data are obtained, the column name corresponding to the column in the second data table is recorded as the first standard name and the keyword, and the field type corresponding to the column in the second database is set to the standard field type.

[0007] Furthermore, the model completed through pre-training and the first column name of the column, obtaining the target number corresponding to the first column name includes:

[0008] The target characteristic word corresponding to the first column name of the column is obtained through the first network layer of the pre-trained first model; and the number corresponding to the target characteristic word is determined to be the target number corresponding to the first column name according to the correspondence between the characteristic words and the numbers saved in advance through the second network layer of the pre-trained first model.

[0009] Furthermore, the model completed through pre-training and the first column name of the column, obtaining the keyword corresponding to the first column name includes:

[0010] The keyword corresponding to the first column name of the column is obtained through the pre-trained second model.

[0011] Furthermore, if the correspondence between the pre-stored serial numbers and the standard data does not include the target serial number, the method further includes:

[0012] Obtain a first field type of data contained in a column corresponding to the first column name in the first data table, and determine the target subject, the first column name and the first field type of the first data table as a second standard name for the column, record the name of the column corresponding to the column in the second data table as the second standard name and the keyword, and set the field type of the column corresponding to the column in the second data table to the first field type.

[0013] Furthermore, the target subject of the first data table is determined by:

[0014] The target topic of the first data table is determined by using a pre-trained third model, the table name of the first data table, and the column name of each column in the first data table.

[0015] Furthermore, the method further comprises:

[0016] Obtain each number in the pre-saved correspondence between numbers and standard data, generate a first number different from each number in the pre-saved correspondence between numbers and standard data, determine that the first field type and the second standard name are second standard data, and add the correspondence between the first number and the second standard data to the correspondence between numbers and standard data.

[0017] Furthermore, the method further comprises:

[0018] For each column of the first data table, if the data contained in the column are all numbers, obtain the target number of different numbers contained in the column, and determine whether there are other columns containing different data with the same number as the target number; if not, determine the third column name corresponding to the second column name of the column based on the correspondence between the pre-saved column names and column names, and add the third column name to the second data table.

[0019] Furthermore, the method further comprises:

[0020] For each column in the second data table, if the fourth column name of the column is the first preset column name, then according to the correspondence between the pre-saved column name and the column name, determine the fifth column name corresponding to the fourth column name, and add the fifth column name to the second data table; if the fourth column name of the column is the second preset column name, then in the correspondence between the pre-saved column name and each field type, determine each candidate field type corresponding to the fourth column name, and for each candidate field type, if the third field type corresponding to the column in the second data table is this candidate field type, take other candidate field types as target field types, add a column with the column name of the fourth column name and the target field type to the second database, and set the field type of the column to the target field type.

[0021] Furthermore, before obtaining the target number and keyword corresponding to the first column name for each column of the first data table to be adjusted through the pre-trained model and the first column name of the column, the method further includes:

[0022] For each column in the first data table, determine whether the column contains data. If so, execute the subsequent operation of obtaining the target number and keyword corresponding to the first column name through the pre-trained model and the first column name of each column in the first data table to be adjusted.

[0023] In a second aspect, an embodiment of the present invention further provides a data processing device, the device comprising:

[0024] an acquisition module, for acquiring, for each column of the first data table to be adjusted, a target number and a keyword corresponding to the first column name through a pre-trained model and the first column name of the column; if the pre-saved correspondence between the number and the standard data includes the target number, acquiring the first standard data corresponding to the target number;

[0025] A processing module is used to establish a second data table corresponding to the first data table, obtain, for each column of the first data table, a first standard name and a standard field type contained in the first standard data, record the column name corresponding to the column in the second data table as the first standard name and the keyword, and set the field type corresponding to the column in the second data table to the standard field type.

[0026] Furthermore, the acquisition module is specifically used to obtain the target characteristic word corresponding to the first column name of the column through the first network layer of the pre-trained first model; and through the second network layer of the pre-trained first model, according to the pre-saved correspondence between the characteristic word and the number, determine that the number corresponding to the target characteristic word is the target number corresponding to the first column name.

[0027] Furthermore, the acquisition module is specifically used to acquire the keyword corresponding to the first column name of the column through a pre-trained second model.

[0028] Furthermore, the processing module is also used to obtain the first field type of the data contained in the column corresponding to the first column name in the first data table, and determine the target subject, the first column name and the first field type of the first data table as the second standard name of the column, record the name of the column corresponding to the column in the second data table as the second standard name and the keyword, and set the field type of the column corresponding to the column in the second database to the first field type.

[0029] Furthermore, the processing module is also used to determine the target topic of the first data table through a pre-trained third model, the table name of the first data table, and the column name of each column in the first data table.

[0030] Furthermore, the processing module is also used to obtain each number in the pre-saved correspondence between numbers and standard data, generate a first number different from each number in the pre-saved correspondence between numbers and standard data, determine the first field type and the second standard name as second standard data, and add the correspondence between the first number and the second standard data to the correspondence between numbers and standard data.

[0031] Furthermore, the processing module is also used to, for each column of the first data table, if the data contained in the column are all numbers, obtain the target number of different numbers contained in the column, and determine whether there are other columns containing different data with the same number as the target number; if not, determine the third column name corresponding to the second column name of the column based on the correspondence between the pre-saved column names and the column names, and add the third column name to the second data table.

[0032] Furthermore, the processing module is also used to, for each column in the second data table, if the fourth column name of the column is the first preset column name, determine the fifth column name corresponding to the fourth column name based on the correspondence between the pre-saved column name and the column name, and add the fifth column name to the second data table; if the fourth column name of the column is the second preset column name, then determine each candidate field type corresponding to the fourth column name in the correspondence between the pre-saved column name and each field type, and for each candidate field type, if the third field type corresponding to the column in the second data table is this candidate field type, use other candidate field types as target field types, add a column with the column name of the fourth column name and the target field type to the second database, and set the field type of the column to the target field type.

[0033] Furthermore, the processing module is also used to determine, for each column in the first data table, whether the column contains data. If so, subsequent operations are performed to obtain, for each column in the first data table to be adjusted, the target number and keyword corresponding to the first column name through the pre-trained model and the first column name of the column.

[0034] In a third aspect, an embodiment of the present invention further provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any of the data processing methods described above when executing a computer program stored in the memory.

[0035] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the data processing methods described above.

[0036] In an embodiment of the present invention, for each column of the first data table to be adjusted, the target number and keyword corresponding to the first column name are obtained through the pre-trained model and the first column name of the column; and the first standard data corresponding to the target number is obtained according to the correspondence between the pre-saved number and the standard data; a second data table corresponding to the first data table is established, and the keyword and the first standard name contained in the first standard data are recorded as the column name corresponding to the column in the second data table, and the field type corresponding to the column in the second data table is set to the standard field type contained in the first standard data. Since in an embodiment of the present invention, for each column in the first data table, the keyword, the first standard name and the standard field type corresponding to the first column name of the column are obtained, and the column name of the column corresponding to the column is recorded as the keyword and the first standard name in the second data table corresponding to the established first data table, the name of each column in the second data table corresponding to the first data table can be standardized, and the field type of the column corresponding to the column in the second data table is set to the standard field type, thereby realizing the standardization of the field type of each column in the second data table corresponding to the first data table. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A schematic diagram of a data processing process provided by an embodiment of the present invention;

[0039] Figure 2 A detailed schematic diagram of a data processing process provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0041] Figure 4 The present invention provides a schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Embodiment 1:

[0044] Figure 1 A schematic diagram of a data processing process provided by an embodiment of the present invention, the process includes the following steps:

[0045] S101: For each column of the first data table to be adjusted, obtain the target number and keyword corresponding to the first column name through the pre-trained model and the first column name of the column; if the correspondence between the pre-saved number and the standard data includes the target number, obtain the first standard data corresponding to the target number.

[0046] The data processing method provided by the embodiment of the present invention is applied to an electronic device, which may be an intelligent device such as a PC or a server.

[0047] In an embodiment of the present invention, in order to adjust the column name of each column in the first data table to be adjusted to a standard name, and in order to adjust the field type of each column in the first data table to a standard field type, for each column in the first data table, the first standard data and keywords corresponding to the first column name of the column are determined. The first standard data includes a first standard name and a standard field type.

[0048] Specifically, for each column in the first data table, the electronic device obtains the name of the first column of the column, and inputs the name of the first column of the column into the pre-trained model to obtain the output of the model, and the output of the model is the target number and keyword corresponding to the name of the first column. The keyword is a word that describes the name of the first column. For example, if the name of the first column is "household registration code", the corresponding keyword may be "household registration", and if the name of the first column is "household registration area division", the corresponding keyword may be "household registration place".

[0049] In an embodiment of the present invention, a correspondence between numbers and standard data is pre-saved in the electronic device. After obtaining the target number corresponding to the first column name, the electronic device determines whether the target number is included in the pre-saved correspondence between numbers and standard data. If the target number is included in the pre-saved correspondence between numbers and standard data, it is determined that the standard data corresponding to the target number is the first standard data.

[0050] Specifically, in an embodiment of the present invention, the electronic device pre-stores a correspondence between table names and data tables. After receiving the table name of the first data table sent by the client, the electronic device determines, based on the received table name of the first data table, that the data table corresponding to the table name is the first data table to be adjusted. In an embodiment of the present invention, each data table stored in the electronic device can be a table in another server obtained by connecting to another server through a preset interface.

[0051] S102: Establish a second data table corresponding to the first data table, and for each column of the first data table, obtain the first standard name and the standard field type contained in the first standard data, record the column name corresponding to the column in the second data table as the first standard name and the keyword, and set the field type corresponding to the column in the second data table to the standard field type.

[0052] In an embodiment of the present invention, the electronic device establishes a second data table corresponding to the first data table, and adjusts the column names and column field types of the second data table. Specifically, the electronic device determines the adjusted column name and the corresponding standard field type for each column based on the determined first standard data and keywords for each column.

[0053] Specifically, the electronic device establishes a second data table corresponding to the first data table, wherein the content of the second data table is completely consistent with the content of the first data table. For each column in the first data table, the first standard name and the standard field type contained in the first standard data corresponding to the column are obtained, wherein the first standard name can be at least one of a Chinese name and an English name, for example, the first standard name is "citizen identity card number", and the standard field types include: string, ins, timestamp, etc.

[0054] For each column in the first data table, after obtaining the first standard name and standard field type corresponding to the column, the electronic device obtains the corresponding column in the second data table, and records the column name of the corresponding column in the second data table as the obtained first standard name and keyword, that is, the column name of the column includes the first standard name and the keyword, and the field type of the corresponding column is set to the standard field type included in the first standard data, thereby standardizing the column name and field type of each column in the second data table corresponding to the first data table.

[0055] In an embodiment of the present invention, for each column in the first data table, the keyword, the first standard name and the standard field type corresponding to the first column name of the column are obtained, and the column name of the column corresponding to the column is recorded as the keyword and the first standard name in the second data table corresponding to the first data table. Therefore, the name of each column in the second data table corresponding to the first data table can be standardized, and the field type of the column corresponding to the column in the second data table can be set to the standard field type, thereby achieving standardization of the field type of each column in the second data table corresponding to the first data table.

[0056] Embodiment 2:

[0057] In order to accurately determine the target number corresponding to each column in the first data table, based on the above embodiment, in an embodiment of the present invention, the model completed through pre-training and the first column name of the column, obtaining the target number corresponding to the first column name includes:

[0058] The target characteristic word corresponding to the first column name of the column is obtained through the first network layer of the pre-trained first model; the target number corresponding to the target characteristic word is determined according to the correspondence between the characteristic word and the number saved in advance through the second network layer of the pre-trained first model.

[0059] In order to accurately determine the target number corresponding to the column name of each column in the first data table, in an embodiment of the present invention, the electronic device determines the target number corresponding to each column in the first data table through a pre-trained first model.

[0060] Specifically, for each column in the first data table, the electronic device inputs the first column name of the column into the pre-trained first model, the input of the first network layer in the first model is the first column name, and the output of the first network layer of the first model is the target characteristic word corresponding to the first column name. The characteristic word is a word representing the characteristics of the column name. For example, if the column name is "where the problem occurs", the corresponding characteristic word may be "problem occurs". The first column name may be a Chinese name or an English name, or may contain both Chinese and English names.

[0061] The first model inputs the acquired target characteristic word into the second network layer of the first model, and the second network layer determines the number corresponding to the target characteristic word as the target number based on the correspondence between the characteristic word and the number saved in advance, and the first model outputs the determined target number. In the embodiment of the present invention, there may be a situation where the target characteristic word determined by the first network layer does not have a corresponding number. If the characteristic words saved in advance in the second network layer do not include the target characteristic word, the preset number is determined as the target number, and the preset number indicates that the target number corresponding to the first column name has not been acquired.

[0062]

[0063]

[0064] Table 1

[0065] Table 1 records the target numbers determined by the first model for different column names. It can be seen from Table 1 that when the column name is "gmsfhm" or "ID card" or contains "gmsfhm and ID card" at the same time, the target number determined is "DE00001"; when the column name is "xm" or "name" or contains "xm and name" at the same time, the target number determined is "DE00002"; when the column name is "jgssxdm" or "household registration area" or contains "jgssxdm and household registration area", the target number determined is "DE00016"; when the column name is "jgssxmc" or "native place" or contains "jgssxmc and native place", the target number determined is "DE00619"; when the column name is "hjd_xzqhdm" or "household registration area" or contains "hjd_xzqhdm and household registration area" at the same time , the target number determined is "DE00070"; when the column name is "hjd_xzqhmc" or "household registration area name" or contains "hjd_xzqhmc and household registration area name", the target number determined is "DE00620"; when the column name is "jgssxdm" or "native place code" or contains "jgssxdm and native place code" at the same time, the target number determined is "DE00016 "; when the column name is "hjd_gajgmc" or "household registration police station" or contains "hjd_gajgmc and household registration police station", the determined target number is "DE00538"; when the column name is "hjd_gajgjgdm" or "household registration police station code" or contains "hjd_gajgjgdm and household registration police station code", the determined target number is "DE00060". In order to accurately determine the keyword corresponding to the column name of each column in the first data table, the model completed by pre-training and the first column name of the column, obtaining the keyword corresponding to the first column name includes:

[0066] The keyword corresponding to the first column name of the column is obtained through the pre-trained second model.

[0067] In the embodiment of the present invention, there may be a situation where the data contained in the column is different but the standard name determined based on the column name is the same, for example, the standard names corresponding to "citizen ID number" and "guardian citizen ID number" are both "citizen ID number", but the data contained in "citizen ID number" and "guardian citizen ID number" are different. Therefore, in the embodiment of the present invention, the adjusted column name of each column in the data table is further determined by keywords.

[0068] In order to determine the keyword corresponding to the column name of each column in the first data table, the electronic device inputs the first column name of each column in the first data table into a pre-trained second model, and obtains the output of the second model, where the output of the second model is the keyword corresponding to the first column name. In an embodiment of the present invention, the determined keyword can be at least one of Chinese or English.

[0069] The specific second model uses natural language processing technology to automatically map the messy text input with the standard keyword table based on the semantic understanding of the neural network to extract keywords. The column names corresponding to each column in the second data table corresponding to the first data table are standardized through the extracted keywords and the standard names contained in the standard data.

[0070] English name Chinese name Keywords jgssxdm Native place code Place of origin jgssxmc Place of origin Household registration area hjd_xzqhdm Household registration area Place of residence hjd_xzqhmc Name of the registered place of residence Detailed address of domicile hjd_gajgmc Police station at the place of residence Police station at the place of residence hjd_gajgjgdm Police station code of the place of residence Current residence area

[0071] Table 2

[0072] Table 2 records the keywords determined by the second model for different column names. It can be seen from Table 2 that when the column name is "jgssxdm" or "native place code" or "jgssxdm and native place code", the keyword determined is "native place"; when the column name is "jgssxmc" or "native place" or "jgssxmc and native place", the keyword determined is "household registration area"; when the column name is "hjd_xzqhdm" or "household registration area" or "hjd_xzqhdm and household registration area", the keyword determined is "household registration place"; When the column name is “hjd_xzqhmc” or “name of the household registration area” or “hjd_xzqhmc and the name of the household registration area”, the keyword determined is “detailed address of household registration”; when the column name is “hjd_gajgmc” or “police station in the household registration area” or “hjd_gajgmc and the police station in the household registration area”, the keyword determined is “police station in the household registration area”; when the column name is “hjd_gajgjgdm” or “code of the police station in the household registration area” or “hjd_gajgjgdm and the code of the police station in the household registration area”, the keyword determined is “current residence area”.

[0073] Embodiment 3:

[0074] In order to accurately determine the field type and name corresponding to each column in the second data table, based on the above embodiments, in an embodiment of the present invention, if the correspondence between the pre-saved numbers and the standard data does not include the target number, the method further includes:

[0075] Obtain a first field type of data contained in a column corresponding to the first column name in the first data table, and determine the target subject, the first column name and the first field type of the first data table as a second standard name for the column, record the name of the column corresponding to the column in the second data table as the second standard name and the keyword, and set the field type of the column corresponding to the column in the second data table to the first field type.

[0076] In an embodiment of the present invention, for each column in the first data table, if the target number determined by the first model is a preset number, then the correspondence between the pre-saved number and the standard data does not contain the target number corresponding to the column, and the corresponding standard data cannot be determined based on the target number, then the standard field type and standard name corresponding to the column are determined respectively.

[0077] Specifically, the electronic device obtains the field type of the data contained in the column corresponding to the first column name in the first data table, determines that the obtained field type is the first field type, and sets the field type of the corresponding column in the second data table to the first field type.

[0078] The electronic device obtains the target subject of the first data table, wherein the correspondence between the table name and the subject of the first data table may be pre-saved in the electronic device, and the electronic device determines the target subject corresponding to the table name of the first data table. After obtaining the target subject of the first data table and the first field type in the column corresponding to the first column name, the electronic device determines that the target subject, the first column name, and the first field type are the second standard name of the column corresponding to the first column name, and determines the keywords corresponding to the second standard name and the first column name as the column name of the column corresponding to the column in the second data table.

[0079] In order to accurately determine the target subject of the first data table, based on the above embodiments, in an embodiment of the present invention, the target subject of the first data table is determined by:

[0080] The target topic of the first data table is determined by using a pre-trained third model, the table name of the first data table, and the column name of each column in the first data table.

[0081] In an embodiment of the present invention, the target subject of the first data table is determined by a pre-trained third model. Specifically, the electronic device inputs the table name of the first data table and the column name of each column in the first data table into the pre-trained third model, and obtains the output of the third model, and the output of the third model is the target subject of the first data table.

[0082] Specifically, the fourth network layer of the third model can extract the characteristic information in each column name and table name, and the fifth network layer of the third model can determine which category the extracted characteristic information belongs to based on the results of the pre-clustering analysis, determine the number of characteristic information belonging to each category, and determine the subject corresponding to the category with a larger number as the target subject of the first data table. And through clustering analysis, the subjects corresponding to the category to which each characteristic word belongs are personnel, offline places, online places, vehicles, Internet terminal equipment, front-end collection equipment, events, information, social units, natural organizations, etc. Specifically, how to cluster the corresponding subjects according to multiple characteristic words is a prior art and will not be repeated here.

[0083] Column Chinese Name Name, age, license plate number, color, case number Column English Name xm, nl, cph, ajbh Table Chinese Name Temporary population, permanent population, vehicle information, KTV information, police information Table English Name zzrk,czrk,clxx

[0084] Table 3

[0085] Table 3 records the different column names and table names input into the third model. It can be seen from Table 3 that the column names input into the third model can be "name, age, license plate number, color, case number", or "xm, nl, cph, ajbh", and the table name can be "temporary population, permanent population, vehicle information, ktv information, police information" or "zzrk, czrk, clxx".

[0086]

[0087]

[0088] Table 4

[0089] Table 4 records different topics and corresponding characteristic information. It can be seen from Table 4 that the class to which the characteristic information "name, gender, ID number, height, weight, age" belongs corresponds to the theme "personnel"; the class to which the characteristic information "license plate number, color, engine, model, vehicle brand" belongs corresponds to the theme "vehicle"; the class to which the characteristic information "case number, case time, case situation" belongs corresponds to the theme "police situation"; the class to which the characteristic information "camera, Internet of Things sensor" belongs corresponds to the theme "front-end acquisition equipment"; the class to which the characteristic information "Internet cafe, KTV, bank" belongs corresponds to the theme "offline place".

[0090] In order to accurately determine the standard data corresponding to each column, based on the above embodiments, in an embodiment of the present invention, the method further includes:

[0091] According to a preset function and each number in the pre-saved correspondence between numbers and standard data, a first number different from each number in the pre-saved correspondence between numbers and standard data is generated, the first field type and the second standard name are determined to be second standard data, and the correspondence between the first number and the second standard data is added to the correspondence between numbers and standard data.

[0092] In an embodiment of the present invention, if the correspondence between the characteristic words and numbers contained in the second network layer of the first model does not include the target characteristic words corresponding to the first column name, then the target number corresponding to the first column name output by the first model is not included in the correspondence between the number and the standard data. Since the first field type and the second standard name corresponding to the first column name have been determined, the electronic device generates a new number, obtains the second standard data composed of the first field and the second standard name, and adds the correspondence between the new number and the second standard data to the correspondence between the number and the standard data pre-saved by the electronic device.

[0093] Specifically, each number in the correspondence between numbers and standard data pre-stored in the electronic device is a number generated in sequence. In order to avoid duplicate numbers in the correspondence between numbers and standard data, in an embodiment of the present invention, the electronic device can obtain the largest number in the correspondence between numbers and standard data, and generate a number larger than the largest number. If the largest number is 0050, the newly generated number can be any integer greater than 0050.

[0094] In addition, in an embodiment of the present invention, the first model can also output the target characteristic word determined by the first network layer, and the electronic device can also obtain the target characteristic word corresponding to the first column name, and add the correspondence between the target characteristic word and the newly generated number to the correspondence between the characteristic word and the number saved in the second network layer of the first model.

[0095] Therefore, in the embodiment of the present invention, the corresponding relationship in the first model can be flexibly increased, so as to ensure the standard names and standard field types of all columns in the determination table.

[0096] Embodiment 4:

[0097] In order to effectively standardize the second data table corresponding to the first data table, based on the above embodiments, in an embodiment of the present invention, the method further includes:

[0098] For each column of the first data table, if the data contained in the column are all numbers, obtain the target number of different numbers contained in the column, and determine whether there are other columns containing different data with the same number as the target number; if not, determine the third column name corresponding to the second column name of the column based on the correspondence between the pre-saved column names and column names, and add the third column name to the second data table.

[0099] This is to prevent the situation where a column in the first data table contains only numbers, but there is no column in the first data table whose data is represented by the numbers, so that the content represented by the numbers cannot be determined based on the numbers contained in the column. For example, the name of a column in the first data table is male and female number, and the column contains only the numbers "0" and "1", and there is no column in the first data table that contains the data of "male" and "female". In an embodiment of the present invention, if the electronic device determines that there is a column in the first data table that contains only numbers, and there is no column in the first data table that contains the data represented by the numbers, the column name of the column that contains only numbers is added to the second data table.

[0100] The electronic device determines, for each column of the first data table, whether the column contains only numbers. If the column contains only numbers, the electronic device determines whether there is a column in the first data table containing data represented by the numbers contained in the column.

[0101] Specifically, when the electronic device determines whether there is a column in the first data table that contains data that is represented by the numbers contained in the column: the electronic device obtains the target number of different numbers contained in the column. For example, if the numbers contained in a column in the first data table are "0, 1, 1, 0, 0", then the different numbers contained in the column in the first data table are "0", "1", and the number of different numbers contained in the column is 2. The electronic device obtains the number of different data contained in other columns in the first data table, and determines whether there is a number of different data contained in other columns that is consistent with the target number of the column. If there is a number of different data contained in other columns that is consistent with the target number, it means that the first data table contains a column, and the data contained in the column is the content of the column that only contains numbers.

[0102] If there is no other column in the first data table containing the same number of different data as the target number, it means that the first data table does not contain a column containing data represented by the numbers in the column. The electronic device determines the third column name corresponding to the second column name of the column in the pre-saved correspondence between column names and column names, adds a new column to the second data table, and determines the column name of the newly added column to be the third column name.

[0103] In an embodiment of the present invention, when determining whether there is a column containing only numbers and whether the data contained is a content column represented by the numbers in the column, if there is a column containing data represented by the numbers in the column, the column containing only numbers can be marked by a first preset code value, and the first preset code value can be any value, such as "1". If there is no column containing data represented by the numbers in the column, the column containing only numbers can be marked by a second preset code value, and the second preset code value can be any value different from the first preset code value, such as "2".

[0104] English name Chinese name Code value tag xbdm Gender Code 1 zdrylbbjdm Key personnel category code 2

[0105] Table 5

[0106] In Table 5, the code value marked as "1" indicates that there is a column containing data represented by the number in the column, and if the code value is marked as "2", it indicates that there is no column containing data represented by the number in the column. From Table 5, we can see that the code value of the column named "xbdm" or "gender code" is marked as 1; the code value of the column named "zdrylbbjdm" or "key personnel category code" is marked as 2.

[0107] Embodiment 5:

[0108] In order to effectively standardize the second data table corresponding to the first data table, based on the above embodiments, in an embodiment of the present invention, the method further includes:

[0109] For each column in the second data table, if the fourth column name of the column is the first preset column name, then according to the correspondence between the pre-saved column name and the column name, determine the fifth column name corresponding to the fourth column name, and add the fifth column name to the second data table; if the fourth column name of the column is the second preset column name, then in the correspondence between the pre-saved column name and each field type, determine each candidate field type corresponding to the fourth column name, and for each candidate field type, if the third field type corresponding to the column in the second data table is this candidate field type, take other candidate field types as target field types, add a column with the column name of the fourth column name and the target field type to the second database, and set the field type of the column to the target field type.

[0110] In the embodiment of the present invention, there may be multiple names for the standard name of a column in the first data table, or there may be multiple types of standard field types in a column, but according to the correspondence between the first model and the number and the standard data, only one standard name and standard field type will be determined. For example, the name of a column standard may be "name" or "name", but according to the correspondence between the first model and the number and the standard data, the determined standard name is "name", so in order to improve the structure of the second data table, for the columns in the first data table that contain multiple standard names or multiple standard field types, columns with other standard names or other standard field types are added in the second data table.

[0111] Specifically, the electronic device pre-stores a first preset column name, and the first preset column name is a column name with multiple standard names. For each column in the second data table, the electronic device determines whether the fourth column name of the column is the pre-stored first preset column name. If the fourth column name of the column is not the first preset column name, it means that the column name of the column is not a column name with multiple standard names; if the fourth column name of the column is the first preset column name, it means that the column name of the column is a column name with multiple standard names. Then, according to the correspondence between the pre-stored column name and the column name, the fifth column name corresponding to the fourth column name is determined, and a column with the fifth column name is added to the second data table. Among them, the fourth column name can be "ID number", and the corresponding fifth column name can be "Citizen ID number".

[0112] In addition, the electronic device pre-stores a second preset column name, and the second preset column name is a column name for which there are multiple standard field types. The electronic device determines whether the fourth column name of each column in the second data table is the second preset column name. If the fourth column name of the column is not the second preset column name, it means that the column name of the column is not a column name for which there are multiple standard field types; if the fourth column name of the column is the second preset column name, it means that the column name of the column is a column name for which there are multiple standard field types, and then determines each candidate field type corresponding to the fourth column name according to the correspondence between the pre-stored column name and each field type. For each determined candidate field type, determine whether the field type corresponding to the column in the second data table is the candidate field type. If the field type corresponding to the column in the second data table is the candidate field type, take the other candidate field types as the target field type, add a column name in the second database with the fourth column name of the column and the target field type, and set the field type of the column to the target field type. Among them, the second preset column name can be a time-related column name. If the data contained in this column in the second data table is November 25, 2021, the data can be expressed as 2021.11.25 or 2021 / 11 / 25 under the target field type.

[0113] In the embodiment of the present invention, if the column name of a column in the first data table is a column name with multiple standard names or multiple standard field types, then corresponding other columns can be accurately added to the second data table. Therefore, in the embodiment of the present invention, the second data table corresponding to the first data table can be effectively standardized.

[0114] In order to effectively standardize the second data table corresponding to the first data table, based on the above embodiments, in an embodiment of the present invention, for each column of the first data table to be adjusted, before obtaining the target number and keyword corresponding to the first column name through the pre-trained model and the first column name of the column, the method further includes:

[0115] For each column in the first data table, determine whether the column contains data. If so, execute the subsequent operation of obtaining the target number and keyword corresponding to the first column name through the pre-trained model and the first column name of each column in the first data table to be adjusted.

[0116] In an embodiment of the present invention, there may be columns containing no data in the first data table. In order to standardize the second data table corresponding to the first data table, the electronic device deletes the columns containing no data in the second data table. That is to say, the second data table established by the electronic device does not contain the columns containing no data in the first data table.

[0117] Specifically, before determining the target number and keyword corresponding to the first column name of each column in the first data table, the electronic device determines whether the column contains data for each column in the first data table. If the column contains data, the subsequent operation is performed, for each column of the first data table to be adjusted, using a pre-trained model and the first column name of the column to obtain the target number and keyword corresponding to the first column name; if the column does not contain data, there is no need to perform the subsequent operation, for each column of the first data table to be adjusted, using a pre-trained model and the first column name of the column to obtain the target number and keyword corresponding to the first column name.

[0118] In addition, in an embodiment of the present invention, for each column in the first data table, after adjusting the column corresponding to the column in the second data table, the electronic device may delete the column for each column in the second data table if there is a corresponding column in the first data table and the column does not contain data.

[0119] Therefore, in the embodiment of the present invention, except for the newly added columns, there are no columns without data in the second data table, so that the second data table corresponding to the first data table can be effectively standardized.

[0120] Embodiment 6:

[0121]

[0122]

[0123] Table 6

[0124] Table 6 records the column name of each column in the second data table, the standard field type in the second data table, the column name and field type of the corresponding column in the first data table, the target number and keyword determined for each column in the second data table, the operation identifier corresponding to each column in the second data table, and the basis for adding the column when the column is a newly added column in the second data table.As can be seen from Table 6, the column name of a certain column in the second data table is "xxzjbh" or "information primary key number" or "xxzjbh and information primary key number", the field type is "string", the column name of the corresponding column in the first data table is "ID" or "id" or "ID and id", the field type of the corresponding column in the first data table is "decimal", the target number corresponding to the column is "DE01115", and the column is a reserved column in the first data table; the column name of a certain column in the second data table is "gmsfhm" or "citizen identity number" or "gmsfhm and citizen identity number", the field type is The type of a column in the second data table is "ccrk_sj" or "initial storage_time" or "ccrk_sj and initial storage_time", the field type of the column in the first data table is "timestamp", and the column name of the column in the first data table is "fir_pet_time". e" or "first entry time" or "fir_pet_tim and first entry time", the field type of the column corresponding to the column in the first data table is "timestamp", the target number corresponding to the column is "DE00102", the keywords determined when determining the column name of the column are "first entry" or "CCRK" or "first entry and CCRK", and the column is a reserved column in the first data table; the column name of a column in the second data table is "ryxl" or "personnel subcategory" or "ryxl and personnel subcategory", the field type is "string", the column is a newly added column, and the basis for adding the column is the first data table. The column name in the table is "pet_second_level"; the column name of a column in the second data table is "cyxjzz" or "participating in illegal organizations" or "cyxjzz and participating in illegal organizations", the field type is "string", the column name of the corresponding column in the first data table is "pet_cult_group" or "participating in illegal organizations" or "pet_cult_group and participating in illegal organizations", the field type of the column corresponding to the column in the first data table is "string", the target number corresponding to the column is "DE50010", and the column is a reserved column in the first data table.

[0125] In addition, in the embodiment of the present invention, the name of the data table after the suffix is ​​removed can also be obtained, and the name after the suffix is ​​removed is determined to be the standard name of the table.

[0126]

[0127] Table 7

[0128] Table 7 is the relevant information of the first data table. It can be seen from Table 7 that the subject of the first data table is "rk (population subject)", the first data table is a data table obtained in the "personnel file creation" system, the "partition (suffix)" in the first data table is "df (daily full volume)", and the standard name of the first data table is "personnel file creation_incoming personnel table" or "dwd_rk_ryjd_rkryb_df", and the table name of the first data table is "ods_personnel file creation_incoming personnel table_df" or "ods_ryjd_rkryb_df".

[0129] Figure 2 A detailed schematic diagram of a data processing process provided by an embodiment of the present invention.

[0130] Depend on Figure 2 It can be seen that the electronic device establishes a connection with other servers through a preset interface and obtains data tables saved by other servers. The specific electronic device can establish a connection with other servers through a user name and password. After establishing a connection with other servers, the electronic device obtains the token of other servers, various projectids and login time.

[0131] The electronic device may first obtain the first data table, and for each column in the first data table, determine the standard name, standard field type and keyword corresponding to the first column name of the column, so as to adjust the name and field type of the column corresponding to the first column name in the second data table corresponding to the first data table. And if the standard name and standard field type are not obtained, the target subject of the first data table is obtained, and the name and field type of the column corresponding to the first column name in the second data table corresponding to the first data table are adjusted according to the target subject and the data contained in the column in the first data table. And determine whether there are columns that need to be added in the second data table, and if there are columns that need to be added, add the corresponding columns.

[0132] After the electronic device has adjusted the second data table, the user cannot edit it. The electronic device creates a new logical model according to the table name, column name and field type in the second data table according to the new logical model creation rules of the management platform, and creates a corresponding physical model through the newly created logical model, so that the second data table can be edited, which is easy to standardize the data management standards in the later stage. And export the table structure for subsequent related use. Specifically, how the electronic device establishes the logical model and the physical model according to the already constructed table is a prior art and will not be repeated here.

[0133] Embodiment 7:

[0134] Figure 3A schematic diagram of a data processing device provided in an embodiment of the present invention, the device comprising:

[0135] The acquisition module 301 is used to acquire, for each column of the first data table to be adjusted, a target number and a keyword corresponding to the first column name through a pre-trained model and the first column name of the column; if the pre-saved correspondence between the number and the standard data includes the target number, then acquire the first standard data corresponding to the target number;

[0136] Processing module 302 is used to establish a second data table corresponding to the first data table, obtain the first standard name and standard field type contained in the first standard data for each column of the first data table, record the column name corresponding to the column in the second data table as the first standard name and the keyword, and set the field type corresponding to the column in the second data table to the standard field type.

[0137] In a possible implementation, the acquisition module 301 is specifically used to acquire the target characteristic word corresponding to the first column name of the column through the first network layer of the pre-trained first model; and to determine, through the second network layer of the pre-trained first model, based on the pre-saved correspondence between the characteristic word and the number, that the number corresponding to the target characteristic word is the target number corresponding to the first column name.

[0138] In a possible implementation, the acquisition module 301 is specifically configured to acquire the keyword corresponding to the first column name of the column through a pre-trained second model.

[0139] In a possible implementation, the processing module 302 is also used to obtain the first field type of the data contained in the column corresponding to the first column name in the first data table, and determine the target subject, the first column name and the first field type of the first data table as the second standard name of the column, record the name of the column corresponding to the column in the second data table as the second standard name and the keyword, and set the field type of the column corresponding to the column in the second database to the first field type.

[0140] In a possible implementation, the processing module 302 is further used to determine the target topic of the first data table through a pre-trained third model, the table name of the first data table, and the column name of each column in the first data table.

[0141] In a possible implementation, the processing module 302 is also used to obtain each number in the pre-saved correspondence between numbers and standard data, generate a first number different from each number in the pre-saved correspondence between numbers and standard data, determine the first field type and the second standard name as second standard data, and add the correspondence between the first number and the second standard data to the correspondence between numbers and standard data.

[0142] In a possible implementation, the processing module 302 is also used to, for each column of the first data table, if the data contained in the column are all numbers, obtain the target number of different numbers contained in the column, determine whether there are other columns containing different data with the same number as the target number, and if not, determine the third column name corresponding to the second column name of the column based on the correspondence between the pre-saved column names and the column names, and add the third column name to the second data table.

[0143] In a possible implementation, the processing module 302 is also used for, for each column in the second data table, if the fourth column name of the column is the first preset column name, then according to the correspondence between the pre-saved column name and the column name, determining the fifth column name corresponding to the fourth column name, and adding the fifth column name to the second data table; if the fourth column name of the column is the second preset column name, then in the correspondence between the pre-saved column name and each field type, determining each candidate field type corresponding to the fourth column name, for each candidate field type, if the third field type corresponding to the column in the second data table is the candidate field type, taking other candidate field types as target field types, adding a column with the column name of the fourth column name and the target field type to the second database, and setting the field type of the column to the target field type.

[0144] In a possible implementation, the processing module 302 is also used to determine, for each column in the first data table, whether the column contains data. If so, a subsequent operation is performed to obtain, for each column in the first data table to be adjusted, the target number and keyword corresponding to the first column name through a pre-trained model and the first column name of the column.

[0145] Embodiment 8:

[0146] Figure 4 FIG. 1 is a schematic diagram of the structure of an electronic device provided by the present invention. Based on the above embodiments, the present invention further provides an electronic device, such as Figure 4As shown, it includes: a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404;

[0147] The memory 403 stores a computer program. When the program is executed by the processor 401, the processor 401 performs the following steps:

[0148] For each column of the first data table to be adjusted, the target number and keyword corresponding to the first column name are obtained through the pre-trained model and the first column name of the column; if the target number is included in the correspondence between the pre-saved number and the standard data, the first standard data corresponding to the target number is obtained; a second data table corresponding to the first data table is established, the first standard name and standard field type included in the first standard data are obtained, the column name corresponding to the column in the second data table is recorded as the first standard name and the keyword, and the field type corresponding to the column in the second database is set to the standard field type.

[0149] In a possible implementation manner, the model completed through pre-training and the first column name of the column, obtaining the target number corresponding to the first column name includes:

[0150] The target characteristic word corresponding to the first column name of the column is obtained through the first network layer of the pre-trained first model; and the number corresponding to the target characteristic word is determined to be the target number corresponding to the first column name according to the correspondence between the characteristic words and the numbers saved in advance through the second network layer of the pre-trained first model.

[0151] In a possible implementation manner, the model completed through pre-training and the first column name of the column, obtaining the keyword corresponding to the first column name includes:

[0152] The keyword corresponding to the first column name of the column is obtained through the pre-trained second model.

[0153] In a possible implementation manner, if the correspondence between the pre-saved serial numbers and the standard data does not include the target serial number, the method further includes:

[0154] Obtain a first field type of data contained in a column corresponding to the first column name in the first data table, and determine the target subject, the first column name and the first field type of the first data table as a second standard name for the column, record the name of the column corresponding to the column in the second data table as the second standard name and the keyword, and set the field type of the column corresponding to the column in the second data table to the first field type.

[0155] In a possible implementation manner, the target subject of the first data table is determined by:

[0156] The target topic of the first data table is determined by using a pre-trained third model, the table name of the first data table, and the column name of each column in the first data table.

[0157] In a possible implementation, the method further includes:

[0158] Obtain each number in the pre-saved correspondence between numbers and standard data, generate a first number different from each number in the pre-saved correspondence between numbers and standard data, determine that the first field type and the second standard name are second standard data, and add the correspondence between the first number and the second standard data to the correspondence between numbers and standard data.

[0159] In a possible implementation, the method further includes:

[0160] For each column of the first data table, if the data contained in the column are all numbers, obtain the target number of different numbers contained in the column, and determine whether there are other columns containing different data with the same number as the target number; if not, determine the third column name corresponding to the second column name of the column based on the correspondence between the pre-saved column names and column names, and add the third column name to the second data table.

[0161] In a possible implementation, the method further includes:

[0162] For each column in the second data table, if the fourth column name of the column is the first preset column name, then according to the correspondence between the pre-saved column name and the column name, determine the fifth column name corresponding to the fourth column name, and add the fifth column name to the second data table; if the fourth column name of the column is the second preset column name, then in the correspondence between the pre-saved column name and each field type, determine each candidate field type corresponding to the fourth column name, and for each candidate field type, if the third field type corresponding to the column in the second data table is this candidate field type, take other candidate field types as target field types, add a column with the column name of the fourth column name and the target field type to the second database, and set the field type of the column to the target field type.

[0163] In a possible implementation manner, before obtaining the target number and keyword corresponding to each column of the first data table to be adjusted by using the pre-trained model and the first column name of the column, the method further includes:

[0164] For each column in the first data table, determine whether the column contains data. If so, execute the subsequent operation of obtaining the target number and keyword corresponding to the first column name through the pre-trained model and the first column name of each column in the first data table to be adjusted.

[0165] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0166] The communication interface is used for communication between the above electronic device and other devices.

[0167] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0168] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0169] Embodiment 9:

[0170] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device, and when the program is run on the electronic device, the electronic device implements the following steps when executing:

[0171] The memory stores a computer program, and when the program is executed by the processor, the processor performs the following steps:

[0172] For each column of the first data table to be adjusted, the target number and keyword corresponding to the first column name are obtained through the pre-trained model and the first column name of the column; if the target number is included in the correspondence between the pre-saved number and the standard data, the first standard data corresponding to the target number is obtained; a second data table corresponding to the first data table is established, the first standard name and standard field type included in the first standard data are obtained, the column name corresponding to the column in the second data table is recorded as the first standard name and the keyword, and the field type corresponding to the column in the second database is set to the standard field type.

[0173] In a possible implementation manner, the model completed through pre-training and the first column name of the column, obtaining the target number corresponding to the first column name includes:

[0174] The target characteristic word corresponding to the first column name of the column is obtained through the first network layer of the pre-trained first model; and the number corresponding to the target characteristic word is determined to be the target number corresponding to the first column name according to the correspondence between the characteristic words and the numbers saved in advance through the second network layer of the pre-trained first model.

[0175] In a possible implementation manner, the model completed through pre-training and the first column name of the column, obtaining the keyword corresponding to the first column name includes:

[0176] The keyword corresponding to the first column name of the column is obtained through the pre-trained second model.

[0177] In a possible implementation manner, if the correspondence between the pre-saved serial numbers and the standard data does not include the target serial number, the method further includes:

[0178] Obtain a first field type of data contained in a column corresponding to the first column name in the first data table, and determine the target subject, the first column name and the first field type of the first data table as a second standard name for the column, record the name of the column corresponding to the column in the second data table as the second standard name and the keyword, and set the field type of the column corresponding to the column in the second data table to the first field type.

[0179] In a possible implementation manner, the target subject of the first data table is determined by:

[0180] The target topic of the first data table is determined by using a pre-trained third model, the table name of the first data table, and the column name of each column in the first data table.

[0181] In a possible implementation, the method further includes:

[0182] Obtain each number in the pre-saved correspondence between numbers and standard data, generate a first number different from each number in the pre-saved correspondence between numbers and standard data, determine that the first field type and the second standard name are second standard data, and add the correspondence between the first number and the second standard data to the correspondence between numbers and standard data.

[0183] In a possible implementation, the method further includes:

[0184] For each column of the first data table, if the data contained in the column are all numbers, obtain the target number of different numbers contained in the column, and determine whether there are other columns containing different data with the same number as the target number; if not, determine the third column name corresponding to the second column name of the column based on the correspondence between the pre-saved column names and column names, and add the third column name to the second data table.

[0185] In a possible implementation, the method further includes:

[0186] For each column in the second data table, if the fourth column name of the column is the first preset column name, then according to the correspondence between the pre-saved column name and the column name, determine the fifth column name corresponding to the fourth column name, and add the fifth column name to the second data table; if the fourth column name of the column is the second preset column name, then in the correspondence between the pre-saved column name and each field type, determine each candidate field type corresponding to the fourth column name, and for each candidate field type, if the third field type corresponding to the column in the second data table is this candidate field type, take other candidate field types as target field types, add a column with the column name of the fourth column name and the target field type to the second database, and set the field type of the column to the target field type.

[0187] In a possible implementation manner, before obtaining the target number and keyword corresponding to each column of the first data table to be adjusted by using the pre-trained model and the first column name of the column, the method further includes:

[0188] For each column in the first data table, determine whether the column contains data. If so, execute the subsequent operation of obtaining the target number and keyword corresponding to the first column name through the pre-trained model and the first column name of each column in the first data table to be adjusted.

[0189] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: For each column of the first data table to be adjusted, the target characteristic word corresponding to the first column name of the column is obtained through the first network layer of the pre-trained first model; the number corresponding to the target characteristic word is determined as the target number corresponding to the first column name through the second network layer of the pre-trained first model according to the correspondence between the characteristic word and the number saved in advance; and the keyword corresponding to the first column name of the column is obtained through the pre-trained second model; if the correspondence between the pre-saved number and the standard data includes the target number, the first standard data corresponding to the target number is obtained; a second data table corresponding to the first data table is established, the first standard name and standard field type included in the first standard data are obtained, the column name corresponding to the column in the second data table is recorded as the first standard name and the keyword, and the field type corresponding to the column in the second database is set to the standard field type.

2. The method according to claim 1, characterized in that If the correspondence between the pre-stored serial numbers and the standard data does not include the target serial number, the method further includes: Obtain a first field type of data contained in a column corresponding to the first column name in the first data table, and determine the target subject, the first column name and the first field type of the first data table as a second standard name for the column, record the name of the column corresponding to the column in the second data table as the second standard name and the keyword, and set the field type of the column corresponding to the column in the second data table to the first field type.

3. The method according to claim 2, characterized in that The target subject of the first data table is determined by: The target topic of the first data table is determined by using a pre-trained third model, the table name of the first data table, and the column name of each column in the first data table.

4. The method according to claim 2, characterized in that: The method further comprises: Obtain each number in the pre-saved correspondence between numbers and standard data, generate a first number different from each number in the pre-saved correspondence between numbers and standard data, determine that the first field type and the second standard name are second standard data, and add the correspondence between the first number and the second standard data to the correspondence between numbers and standard data.

5. The method according to claim 1, characterized in that The method further comprises: For each column of the first data table, if the data contained in the column are all numbers, obtain the target number of different numbers contained in the column, and determine whether there are other columns containing different data with the same number as the target number; if not, determine the third column name corresponding to the second column name of the column based on the correspondence between the pre-saved column names and column names, and add the third column name to the second data table.

6. The method according to claim 1, characterized in that The method further comprises: For each column in the second data table, if the fourth column name of the column is the first preset column name, then according to the correspondence between the pre-saved column name and the column name, determine the fifth column name corresponding to the fourth column name, and add the fifth column name to the second data table; if the fourth column name of the column is the second preset column name, then in the correspondence between the pre-saved column name and each field type, determine each candidate field type corresponding to the fourth column name, and for each candidate field type, if the third field type corresponding to the column in the second data table is this candidate field type, take other candidate field types as target field types, add a column with the column name of the fourth column name and the target field type to the second database, and set the field type of the column to the target field type.

7. The method according to claim 1, characterized in that Before obtaining the target number and keyword corresponding to the first column name for each column of the first data table to be adjusted by using the pre-trained model and the first column name of the column, the method further includes: For each column in the first data table, determine whether the column contains data. If so, execute the subsequent operation of obtaining the target number and keyword corresponding to the first column name through the pre-trained model and the first column name of each column in the first data table to be adjusted.

8. A data processing device, characterized in that: The device comprises: An acquisition module is used to acquire, for each column of the first data table to be adjusted, a target characteristic word corresponding to the first column name of the column through the first network layer of the pre-trained first model; determine, through the second network layer of the pre-trained first model, based on the correspondence between the characteristic word and the number saved in advance, that the number corresponding to the target characteristic word is the target number corresponding to the first column name; and acquire, through the pre-trained second model, a keyword corresponding to the first column name of the column; if the correspondence between the pre-saved number and the standard data includes the target number, then acquire the first standard data corresponding to the target number; A processing module is used to establish a second data table corresponding to the first data table, obtain, for each column of the first data table, a first standard name and a standard field type contained in the first standard data, record the column name corresponding to the column in the second data table as the first standard name and the keyword, and set the field type corresponding to the column in the second data table to the standard field type.

9. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the data processing method as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 7.

Images