Code field recognition method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a code field recognition method, device, electronic device and storage medium. The embodiment of the present invention obtains statistical data by counting the values ​​of the target field of the data table. If the statistical data meets the preset conditions, the characteristic value of the preset field feature corresponding to the target field is obtained according to the statistical data, the characteristic value is input into a trained code field recognition model, and whether the target field is a code field is determined according to the output result of the code field recognition model, thereby reducing manual input and improving the efficiency of code field recognition.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a code field recognition method, device, electronic device and storage medium. Background Art

[0002] In the process of building a big data platform for government affairs and enterprises, it is necessary to extract a large number of business system tables from numerous business systems into the data warehouse, and then clean and process the business data in the table to make it a reusable data resource. In the process of data governance, it is an important task to identify the code field in the business system table. Only by identifying the code field can the data quality of the field be judged, and then processed to form key and standard data resources.

[0003] In the related art, the code field is identified manually, which is inefficient. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present invention provides a code field recognition method, device, electronic device and storage medium to improve the efficiency of code field recognition.

[0005] According to a first aspect of an embodiment of the present invention, a code field identification method is provided, the method comprising:

[0006] Count the values ​​of the target fields in the data table to obtain statistical data;

[0007] If the statistical data meets the preset condition, obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data;

[0008] The feature value is input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

[0009] According to a second aspect of an embodiment of the present invention, a code field recognition device is provided, the device comprising:

[0010] The statistics module is used to count the values ​​of the target fields in the data table and obtain statistical data;

[0011] A feature value obtaining module, used for obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data if the statistical data meets a preset condition;

[0012] The determination module is used to input the feature value into a trained code field recognition model, and determine whether the target field is a code field according to the output result of the code field recognition model.

[0013] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a processor and a memory for storing executable instructions of the processor;

[0014] The processor is configured to:

[0015] Count the values ​​of the target fields in the data table to obtain statistical data;

[0016] If the statistical data meets the preset condition, obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data;

[0017] The feature value is input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

[0018] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein a plurality of computer instructions are stored on the computer-readable storage medium, and when the computer instructions are executed, the following processing is performed:

[0019] Count the values ​​of the target fields in the data table to obtain statistical data;

[0020] If the statistical data meets the preset condition, obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data;

[0021] The feature value is input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

[0022] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0023] In an embodiment of the present invention, statistical data are obtained by counting the values ​​of the target fields of the data table. If the statistical data meets a preset condition, a feature value of a preset field feature corresponding to the target field is obtained based on the statistical data, and the feature value is input into a trained code field recognition model. Based on the output result of the code field recognition model, it is determined whether the target field is a code field, thereby reducing manual input and improving the efficiency of code field recognition.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0026] Figure 1 It is an example flow chart of the code field identification method provided in an embodiment of the present invention.

[0027] Figure 2 It is a functional block diagram of a code field recognition device provided by an embodiment of the present invention.

[0028] Figure 3 It is a hardware structure diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0030] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments of the present invention, and are not intended to limit the embodiments of the present invention. The singular forms of "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, third, etc. may be used to describe various information in embodiments of the present invention, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] In a business system, if a business field stores a value with a limited range of numerical values ​​or codes, we call such a field a code field. For example, the value of the Chinese ethnic group cannot exceed 56, otherwise it can be regarded as abnormal data.

[0033] In the process of data governance, it is very important to identify the code fields and perform standardized cleaning of the data. In the related art, the code fields are identified manually.

[0034] One of the ways is that the data architect confirms the code field by communicating with the builder of the business system.

[0035] Another way is that the data architect analyzes the records in the database himself and judges the code fields based on his own experience.

[0036] From the above two methods, it can be seen that the disadvantages of manual identification of code fields are obvious, that is, it requires a lot of human energy and time, and the workload is very huge. This not only makes the manpower cost required for code field identification extremely high, but also takes a long time and is inefficient.

[0037] The code field recognition method provided by the present invention is described in detail below through embodiments.

[0038] Figure 1 FIG. 1 is a flowchart of a code field recognition method provided by an embodiment of the present invention. Figure 1 As shown, the code field identification method may include:

[0039] S101, counting the values ​​of the target fields in the data table to obtain statistical data.

[0040] S102: If the statistical data meets a preset condition, obtain a feature value of a preset field feature corresponding to the target field according to the statistical data.

[0041] S103, inputting the feature value into a trained code field recognition model, and determining whether the target field is a code field according to an output result of the code field recognition model.

[0042] In this embodiment, the target field is a field that needs to be identified by a code field. For example, when it is necessary to identify a code field from data table 1, each field in data table 1 can be identified separately according to the code field identification method provided in this embodiment. At this time, each field in the data table is a target field.

[0043] In applications, the code field recognition method provided in this embodiment may also be used to perform batch recognition on fields in multiple data tables.

[0044] In one example, the statistical data may include the number of statistical records, vacancy rate, and code distribution.

[0045] The number of statistical records refers to the number of records sampled in the statistics.

[0046] Usually, the number of records in a data table is very large. To improve efficiency, you can sample the records of the field and then perform statistics. For example, if data table 1 has 1 million records, then each field in data table 1 has 1 million records, and 10,000 records are sampled from the 1 million records of the field for statistics.

[0047] The vacancy rate refers to the ratio of the number of records with empty field values ​​to the number of statistical records.

[0048] Here, code distribution refers to the number of each code.

[0049] The statistical data of a certain field can be shown in Table 1.

[0050] Table 1

[0051]

[0052] Among them, the record number (record_num) in Table 1 is the number of statistical records. In Table 1, the code distribution indicates that in the 2092 records of the statistical field "xzbg", there are 100 dictionary values ​​(i.e., the keys in the key (Key) value (Value) pair), of which there are 61 records with a dictionary value of 15, 260 records with a dictionary value of 20, 500 records with a dictionary value of 10, and 471 records with a dictionary value of 25.

[0053] In an example, the preset condition may be that the vacancy rate of the value of the target field in the statistical result is less than a preset vacancy rate threshold, and the code distribution data of the target field in the statistical result is not empty.

[0054] If the vacancy rate in the statistical data of a certain field is greater than or equal to the vacancy rate threshold, or the code distribution data of the field is empty, the statistical data of the field can be deleted and the code field will not be identified because the statistical data of the field has little available information or cannot provide valid information.

[0055] In an example, the preset field features may include the number of statistical records of the field, the number of code types, the maximum and minimum values ​​of the number of occurrences of each code, and the actual number of records.

[0056] Whether it is a code field is positively correlated with the concentration of code distribution, and the concentration of code distribution is positively correlated with the ratio of the number of records to the number of code types. Therefore, the number of code types for each field can be counted as a preset field feature.

[0057] The frequency of occurrence of each code and the maximum and minimum values ​​of the number of code occurrences in the field to which the code belongs can also reflect the concentration of code distribution to a certain extent. Therefore, the maximum and minimum values ​​of the number of code occurrences in the field are also used as preset field features.

[0058] Due to the limitation of the number of words in the table, some code distributions contain incomplete statistical key-value pairs, so the number of statistical records is not the number of records in the actual table. Therefore, the actual number of existing records (real_record_num) can be counted based on the code distribution.

[0059] The actual number of records is equal to the product of the number of statistical records and the first ratio, where the first ratio is the ratio of the actual total number of records in the field to the number of records sampled in the statistics. For example, assuming that the actual total number of records in the field is 1 million, the number of records sampled in the statistics is 10,000, and the number of statistical records is 2,000, then the actual number of records is equal to 2,000 × (100 ÷ 1) = 200,000.

[0060] In an exemplary implementation process, obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data may include:

[0061] Cleaning the statistical data to obtain target data;

[0062] According to the target data, a feature value of a preset field feature corresponding to the target field is determined.

[0063] There may be a lot of noise data caused by human operations in the code distribution, such as inconsistent uppercase and lowercase letters, Chinese typos, simultaneous appearance of names and codes, etc. Therefore, the key-value pair data in the code distribution is cleaned to remove long-tail data and non-standard data to form standardized records.

[0064] Through cleaning, more standardized target data can be obtained, so that the characteristic values ​​of preset field features can be determined based on the target data, which can improve the accuracy of code field recognition.

[0065] In one example, the process of obtaining the code field identification model may include:

[0066] Setting up the machine learning model;

[0067] Acquire sample data, where the sample data includes a feature value of a preset field feature corresponding to a sample field and a label value corresponding to the sample field, where the label value is used to indicate whether the sample field is a code field;

[0068] The machine learning model is trained using the sample data to obtain a trained machine learning model, and the trained machine learning model is used as a code field recognition model.

[0069] Among them, the characteristic values ​​of each preset field feature corresponding to the sample field can be obtained through the statistical data of the sample field. The label value corresponding to the sample field can be manually marked. For example, a label value of 1 indicates that it is a code field, and a label value of 0 indicates that it is not a code field. If it is known that sample field 1 is a code field and sample field 2 is not a code field, then the label value corresponding to sample field 1 is 1, and the label value corresponding to sample field 2 is 0.

[0070] In one example, the machine learning model may be a binary classification model, such as an XGBoost (Extreme Gradient Boosting) algorithm model. In other examples, the machine learning model may also use other classifier models, which is not limited in this embodiment.

[0071] In one example, the method may further include:

[0072] Obtain a target word vector corresponding to the target field;

[0073] Inputting the feature value into a trained code field recognition model, and determining whether the target field is a code field according to an output result of the code field recognition model, including:

[0074] The feature value and the target word vector are input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

[0075] In an example, obtaining a target word vector corresponding to the target field may include:

[0076] According to the field name of the target field, a target word vector corresponding to the target field is determined.

[0077] The field name of the target field may be obtained from the statistical data of the target field (as shown in the aforementioned Table 1).

[0078] In an example, obtaining a target word vector corresponding to the target field may include:

[0079] Determine a target word vector corresponding to the target field according to the field description information of the target field.

[0080] The field description information of the target field may be obtained from the statistical data of the target field. For example, the value of the "field description" field in the aforementioned Table 1 is the field description information.

[0081] In this embodiment, a word vector database can be preset in the system, and the target word vector corresponding to the target field can be obtained by querying the word vector database. For example, the query condition can be the field name or field description information of the target field.

[0082] Different word vector databases can be set up in different scenarios according to application needs. In the same scenario, the word vector database can also be updated as more empirical data is accumulated.

[0083] In this embodiment, word vectors are also used as inputs of the code field recognition model, which enriches the input features of the code field recognition model and can improve the performance of the code field recognition model.

[0084] When training the code field recognition model of this embodiment, the sample data includes not only the feature value of the preset field feature corresponding to the sample field and the label value corresponding to the sample field, but also the word vector corresponding to the sample field.

[0085] In one example, the process of obtaining a code field recognition model including word vectors in input data may include:

[0086] Setting up the machine learning model;

[0087] Acquire sample data, where the sample data includes a feature value of a preset field feature corresponding to a sample field, a word vector corresponding to the sample field, and a label value corresponding to the sample field, where the label value is used to indicate whether the sample field is a code field;

[0088] The machine learning model is trained using the sample data to obtain a trained machine learning model, and the trained machine learning model is used as a code field recognition model.

[0089] The characteristic values ​​of each preset field feature corresponding to the sample field can be obtained through the statistical data of the sample field. The word vector corresponding to the sample field can be obtained by querying the word vector database according to the field name or field description information of the sample field. The label value corresponding to the sample field can be manually marked.

[0090] The code field recognition method provided by the embodiment of the present invention is further described in detail below by using examples.

[0091] Assuming there are 100 fields, field 1 to field 100, the code field recognition process for field 1 to field 100 can be as follows:

[0092] (1) Count the values ​​of field 1 to field 100 respectively to obtain corresponding statistical data 1 to statistical data 100.

[0093] (2) Assuming that the vacancy rate threshold is 90%, where the vacancy rates of fields 1 to 5 are greater than 90%, statistical data 1 to statistical data 5 corresponding to fields 1 to 5 are deleted, and statistical data 6 to statistical data 100 remain.

[0094] (3) Statistical data 6 to statistical data 100 are cleaned respectively, and long-tail data and irregular data in statistical data 6 to statistical data 100 are deleted to obtain target data 6 to target data 100.

[0095] (4) According to target data 6 to target data 100, obtain the statistical record number, code type number, maximum and minimum values ​​of the number of occurrences of each code, actual record number and word vector corresponding to fields 6 to 100 respectively.

[0096] (5) Determine whether fields 6 to 100 are code fields, respectively, based on the statistical record number, code type number, maximum and minimum values ​​of the number of occurrences of each code, actual record number, and word vector corresponding to fields 6 to 100. Taking field 6 as an example, the statistical record number, code type number, maximum and minimum values ​​of the number of occurrences of each code, actual record number, and word vector corresponding to field 6 are input into the trained code field recognition model. If the output result of the code field recognition model is 1, it is determined that field 6 is a code field. If the output result of the code field recognition model is 0, it is determined that field 6 is not a code field.

[0097] It can be seen that the embodiment of the present invention can automatically and batch-wise perform code field recognition, which greatly improves the efficiency of code field recognition.

[0098] In one example, after step S103, the following steps may also be included:

[0099] If it is determined that the target field is a code field, a dictionary table is generated according to the dictionary name and the corresponding dictionary value contained in the target field.

[0100] If there is no existing dictionary table, you can

[0101] The dictionary table records the key (dictionary name) value (dictionary value) pairs in the code field, which can be used as a standard to clean the data table to be cleaned.

[0102] Each record in the dictionary table may include information such as the dictionary table name, dictionary table ID, standard dictionary name, and standard dictionary value.

[0103] In one example, generating a dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field may include:

[0104] Determine the dictionary table name according to the field description information of the target field;

[0105] According to the dictionary name and the corresponding dictionary value of the target field, a standard dictionary name and the corresponding standard dictionary value are added to the dictionary table.

[0106] In one example, the method may further include:

[0107] If it is determined that the target field is a code field, the existing dictionary table is modified according to the dictionary name and the corresponding dictionary value contained in the target field.

[0108] In the case where a dictionary table already exists, there is no need to generate a dictionary table, and the dictionary name contained in the target field can be added to the existing dictionary table by adding a dictionary table record.

[0109] In one example, modifying an existing dictionary table according to a dictionary name and a corresponding dictionary value included in the target field may include:

[0110] If the existing dictionary table contains the dictionary table name corresponding to the target field, and the target field contains a target dictionary name that is not contained in the existing dictionary table, the target dictionary name and the corresponding dictionary value are added to the existing dictionary table.

[0111] For example, assume that the code field is as shown in Table 2 and the existing dictionary table is as shown in Table 3.

[0112] Table 2

[0113]

[0114]

[0115] Table 3

[0116] Category Dictionary table name Dictionary table ID Standard dictionary name Standard dictionary values public job title aca10ef8-beb3-4068-8e71-bd0b4d973baf Assistant Engineer 10 public job title aca10ef8-beb3-4068-8e71-bd0b4d973baf researcher 40 public job title aca10ef8-beb3-4068-8e71-bd0b4d973baf Senior Engineer 30 public job title aca10ef8-beb3-4068-8e71-bd0b4d973baf engineer 20

[0117] According to Table 2 and Table 3, the "title" field contains 13 dictionary names, of which 4 dictionary names "assistant engineer", "researcher", "senior engineer", and "engineer" are also included in the existing dictionary table shown in Table 3, and 9 dictionary names (such as "associate professor", "associate researcher", etc.) are not included in the existing dictionary table shown in Table 3. At this time, these 9 dictionary names and their corresponding dictionary values ​​are added to the existing dictionary table shown in Table 3.

[0118] In some cases, the code field contains dictionary names and dictionary values. In this case, you can extract the dictionary value corresponding to the dictionary name from the code field. When the code field only contains the dictionary name but not the dictionary value, you can manually set the dictionary value for the dictionary name. When setting, you need to ensure that the dictionary values ​​for different dictionary names are also different.

[0119] In one example, modifying an existing dictionary table according to a dictionary name and a corresponding dictionary value included in the target field may include:

[0120] If the existing dictionary table contains the dictionary table name corresponding to the target field, and the target field contains a target dictionary name not included in the existing dictionary table, and if the distribution quantity of the target dictionary name in the target field meets the preset centralized distribution condition, the target dictionary name and the corresponding dictionary value are added to the existing dictionary table.

[0121] Among them, satisfying the concentrated distribution condition indicates that the corresponding dictionary names are distributed more.

[0122] In an example, the concentrated distribution condition may be: a ratio of the distribution quantity of the target dictionary names in the target field to the sum of the distribution quantities of all dictionary names in the target field is greater than or equal to a preset percentage.

[0123] In another example, the concentrated distribution condition may be: the distribution quantity of the target dictionary name in the target field is greater than or equal to a preset quantity threshold.

[0124] For example, according to Table 3, in the code distribution of the "title" field, the dictionary names are mainly concentrated in "researcher", "associate researcher", "professor", "associate professor", "assistant engineer", "engineer" and "senior engineer". Among them, four dictionary names "assistant engineer", "researcher", "senior engineer" and "engineer" are already included in the dictionary table shown in Table 3. According to this embodiment, only the three dictionary names "associate researcher", "professor" and "associate professor" and the corresponding dictionary values ​​need to be added to the dictionary table shown in Table 3.

[0125] In one example, modifying an existing dictionary table according to a dictionary name and a corresponding dictionary value included in the target field may include:

[0126] If the existing dictionary table does not contain the dictionary table name corresponding to the target field, all dictionary names and corresponding dictionary values ​​contained in the target field are added to the existing dictionary table.

[0127] When the existing dictionary table does not contain the dictionary table name corresponding to the target field, it can be directly determined that the dictionary names in the target field are not included in the existing dictionary table. At this time, there is no need to judge each dictionary name in the target field, which can save processing time.

[0128] In one example, modifying an existing dictionary table according to a dictionary name and a corresponding dictionary value included in the target field may include:

[0129] If the first dictionary name contained in the target field is the same as the second dictionary name in the existing dictionary table, but the first dictionary value corresponding to the first dictionary name in the target field is different from the second dictionary value corresponding to the second dictionary name in the existing dictionary table, modify the second dictionary value in the existing dictionary table to the first dictionary value.

[0130] In this embodiment, the first dictionary name and the second dictionary name are the same dictionary name, and different names are used here to distinguish the locations of the dictionary names (target fields or existing dictionary tables).

[0131] In some cases, according to the new data standard, the dictionary values ​​corresponding to some dictionary names will be updated, that is, the new dictionary values ​​are different from the old dictionary values. When the dictionary table is established earlier, and the data table containing the target field is established later than the dictionary table, according to the new data standard, the dictionary value used in the target field is the new dictionary value, and the corresponding dictionary value in the existing dictionary table is the old dictionary value. At this time, in order to make the dictionary table meet the new data standard and continue to be used in data cleaning scenarios, the old dictionary value needs to be changed to the new dictionary value.

[0132] In an example, before modifying the second dictionary value in the existing dictionary table to the first dictionary value, the following may also be included:

[0133] Display the prompt information for modifying the dictionary value and the information to confirm whether to modify it;

[0134] Modifying the second dictionary value in the existing dictionary table to the first dictionary value includes:

[0135] If a message confirming the modification is received, the second dictionary value in the existing dictionary table is modified to the first dictionary value.

[0136] The displayed confirmation information of whether to modify may be a clickable virtual button such as "OK" (indicating agreement to modify), "Cancel" (indicating refusal to modify), or "Yes" (indicating agreement to modify), "No" (indicating refusal to modify). The user may indicate his / her subjective will of whether to agree to modify through the confirmation information of whether to modify. For example, the user may click the virtual button "OK" to agree to modify, or click the virtual button "Cancel" to refuse to modify.

[0137] The code field recognition method provided by the embodiment of the present invention obtains statistical data by counting the values ​​of the target field of the data table. If the statistical data meets the preset conditions, the characteristic value of the preset field feature corresponding to the target field is obtained according to the statistical data, and the characteristic value is input into the trained code field recognition model. According to the output result of the code field recognition model, it is determined whether the target field is a code field, which reduces the manual input and improves the efficiency of code field recognition. At the same time, due to the reduction of manual input, the present embodiment also greatly reduces the service cost of code field recognition.

[0138] Based on the above method embodiments, the present invention also provides corresponding device, equipment and storage medium embodiments. For detailed implementation of the device, equipment and storage medium embodiments of the present invention, please refer to the corresponding description of the above method embodiment part.

[0139] Figure 2 is a functional block diagram of a code field recognition device provided by an embodiment of the present invention. Figure 2 As shown, in this embodiment, the code field identification device may include:

[0140] The statistics module 210 is used to count the values ​​of the target fields of the data table to obtain statistical data;

[0141] A feature value obtaining module 220 is used to obtain a feature value of a preset field feature corresponding to the target field according to the statistical data if the statistical data meets a preset condition;

[0142] The determination module 230 is used to input the feature value into a trained code field recognition model, and determine whether the target field is a code field according to the output result of the code field recognition model.

[0143] In one example, the feature value obtaining module 220 may be specifically used for:

[0144] Cleaning the statistical data to obtain target data;

[0145] According to the target data, a feature value of a preset field feature corresponding to the target field is determined.

[0146] In one example, the process of obtaining the code field identification model includes:

[0147] Setting up the machine learning model;

[0148] Acquire sample data, where the sample data includes a feature value of a preset field feature corresponding to a sample field and a label value corresponding to the sample field, where the label value is used to indicate whether the sample field is a code field;

[0149] The machine learning model is trained using the sample data to obtain a trained machine learning model, and the trained machine learning model is used as a code field recognition model.

[0150] In one example, the method further includes:

[0151] Obtain a target word vector corresponding to the target field;

[0152] Inputting the feature value into a trained code field recognition model, and determining whether the target field is a code field according to an output result of the code field recognition model, including:

[0153] The feature value and the target word vector are input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

[0154] In one example, the preset field characteristics include the statistical record number of the field, the number of code types, the maximum and minimum values ​​of the number of occurrences of each code, and the actual number of records; wherein the actual number of records is equal to the product of the statistical record number and a first ratio, and the first ratio is the ratio of the actual total number of records of the target field to the number of records sampled in the statistics.

[0155] In one example, the statistical data includes the number of statistical records, vacancy rate, and code distribution.

[0156] In one example, the preset condition is that the vacancy rate of the value of the target field in the statistical result is less than a preset vacancy rate threshold, and the code distribution data of the target field in the statistical result is not empty.

[0157] In one example, it also includes:

[0158] The dictionary table generating module is used to generate a dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field if it is determined that the target field is a code field.

[0159] In one example, it also includes:

[0160] The dictionary table modification module is used to modify the existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field if it is determined that the target field is a code field.

[0161] In one example, the dictionary table modification module may be specifically used to:

[0162] If the existing dictionary table contains the dictionary table name corresponding to the target field, and the target field contains a target dictionary name that is not contained in the existing dictionary table, the target dictionary name and the corresponding dictionary value are added to the existing dictionary table.

[0163] In one example, the dictionary table modification module may be specifically used to:

[0164] If the existing dictionary table does not contain the dictionary table name corresponding to the target field, all dictionary names and corresponding dictionary values ​​contained in the target field are added to the existing dictionary table.

[0165] In one example, the dictionary table modification module may be specifically used to:

[0166] If the first dictionary name contained in the target field is the same as the second dictionary name in the existing dictionary table, but the first dictionary value corresponding to the first dictionary name in the target field is different from the second dictionary value corresponding to the second dictionary name in the existing dictionary table, modify the second dictionary value in the existing dictionary table to the first dictionary value.

[0167] In one example, it also includes:

[0168] A display module, used for displaying prompt information for modifying the dictionary value and information for confirming whether to modify before modifying the second dictionary value in the existing dictionary table to the first dictionary value;

[0169] When the dictionary table modification module is used to modify the second dictionary value in the existing dictionary table to the first dictionary value, it can be specifically used to:

[0170] If a message confirming the modification is received, the second dictionary value in the existing dictionary table is modified to the first dictionary value.

[0171] An embodiment of the present invention further provides an electronic device. Figure 3 FIG. 1 is a hardware structure diagram of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, the electronic device includes: an internal bus 301, and a memory 302, a processor 303 and an external interface 304 connected via the internal bus.

[0172] When the electronic device acts as a working node, the processor 303 is used to read the machine-readable instructions on the memory 302 and execute the instructions to implement the following operations:

[0173] Count the values ​​of the target fields in the data table to obtain statistical data;

[0174] If the statistical data meets the preset condition, obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data;

[0175] The feature value is input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

[0176] In one example, obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data includes:

[0177] Cleaning the statistical data to obtain target data;

[0178] According to the target data, a feature value of a preset field feature corresponding to the target field is determined.

[0179] In one example, the process of obtaining the code field identification model includes:

[0180] Setting up the machine learning model;

[0181] Acquire sample data, where the sample data includes a feature value of a preset field feature corresponding to a sample field and a label value corresponding to the sample field, where the label value is used to indicate whether the sample field is a code field;

[0182] The machine learning model is trained using the sample data to obtain a trained machine learning model, and the trained machine learning model is used as a code field recognition model.

[0183] In one example, it also includes:

[0184] Obtain a target word vector corresponding to the target field;

[0185] Inputting the feature value into a trained code field recognition model, and determining whether the target field is a code field according to an output result of the code field recognition model, including:

[0186] The feature value and the target word vector are input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

[0187] In one example, the preset field characteristics include the statistical record number of the field, the number of code types, the maximum and minimum values ​​of the number of occurrences of each code, and the actual number of records; wherein the actual number of records is equal to the product of the statistical record number and a first ratio, and the first ratio is the ratio of the actual total number of records of the target field to the number of records sampled in the statistics.

[0188] In one example, the statistical data includes the number of statistical records, vacancy rate, and code distribution.

[0189] In one example, the preset condition is that the vacancy rate of the value of the target field in the statistical result is less than a preset vacancy rate threshold, and the code distribution data of the target field in the statistical result is not empty.

[0190] In one example, it also includes:

[0191] If it is determined that the target field is a code field, a dictionary table is generated according to the dictionary name and the corresponding dictionary value contained in the target field.

[0192] In one example, it also includes:

[0193] If it is determined that the target field is a code field, the existing dictionary table is modified according to the dictionary name and the corresponding dictionary value contained in the target field.

[0194] In one example, modifying an existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field includes:

[0195] If the existing dictionary table contains the dictionary table name corresponding to the target field, and the target field contains a target dictionary name that is not contained in the existing dictionary table, the target dictionary name and the corresponding dictionary value are added to the existing dictionary table.

[0196] In one example, modifying an existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field includes:

[0197] If the existing dictionary table does not contain the dictionary table name corresponding to the target field, all dictionary names and corresponding dictionary values ​​contained in the target field are added to the existing dictionary table.

[0198] In one example, modifying an existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field includes:

[0199] If the first dictionary name contained in the target field is the same as the second dictionary name in the existing dictionary table, but the first dictionary value corresponding to the first dictionary name in the target field is different from the second dictionary value corresponding to the second dictionary name in the existing dictionary table, modify the second dictionary value in the existing dictionary table to the first dictionary value.

[0200] In one example, before modifying the second dictionary value in the existing dictionary table to the first dictionary value, the method further includes:

[0201] Display the prompt information for modifying the dictionary value and the information to confirm whether to modify it;

[0202] Modifying the second dictionary value in the existing dictionary table to the first dictionary value includes:

[0203] If a message confirming the modification is received, the second dictionary value in the existing dictionary table is modified to the first dictionary value.

[0204] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed, the following processing is performed:

[0205] Count the values ​​of the target fields in the data table to obtain statistical data;

[0206] If the statistical data meets the preset condition, obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data;

[0207] The feature value is input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

[0208] In one example, obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data includes:

[0209] Cleaning the statistical data to obtain target data;

[0210] According to the target data, a feature value of a preset field feature corresponding to the target field is determined.

[0211] In one example, the process of obtaining the code field identification model includes:

[0212] Setting up the machine learning model;

[0213] Acquire sample data, where the sample data includes a feature value of a preset field feature corresponding to a sample field and a label value corresponding to the sample field, where the label value is used to indicate whether the sample field is a code field;

[0214] The machine learning model is trained using the sample data to obtain a trained machine learning model, and the trained machine learning model is used as a code field recognition model.

[0215] In one example, it also includes:

[0216] Obtain a target word vector corresponding to the target field;

[0217] Inputting the feature value into a trained code field recognition model, and determining whether the target field is a code field according to an output result of the code field recognition model, including:

[0218] The feature value and the target word vector are input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

[0219] In one example, the preset field characteristics include the statistical record number of the field, the number of code types, the maximum and minimum values ​​of the number of occurrences of each code, and the actual number of records; wherein the actual number of records is equal to the product of the statistical record number and a first ratio, and the first ratio is the ratio of the actual total number of records of the target field to the number of records sampled in the statistics.

[0220] In one example, the statistical data includes the number of statistical records, vacancy rate, and code distribution.

[0221] In one example, the preset condition is that the vacancy rate of the value of the target field in the statistical result is less than a preset vacancy rate threshold, and the code distribution data of the target field in the statistical result is not empty.

[0222] In one example, it also includes:

[0223] If it is determined that the target field is a code field, a dictionary table is generated according to the dictionary name and the corresponding dictionary value contained in the target field.

[0224] In one example, it also includes:

[0225] If it is determined that the target field is a code field, the existing dictionary table is modified according to the dictionary name and the corresponding dictionary value contained in the target field.

[0226] In one example, modifying an existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field includes:

[0227] If the existing dictionary table contains the dictionary table name corresponding to the target field, and the target field contains a target dictionary name that is not contained in the existing dictionary table, the target dictionary name and the corresponding dictionary value are added to the existing dictionary table.

[0228] In one example, modifying an existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field includes:

[0229] If the existing dictionary table does not contain the dictionary table name corresponding to the target field, all dictionary names and corresponding dictionary values ​​contained in the target field are added to the existing dictionary table.

[0230] In one example, modifying an existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field includes:

[0231] If the first dictionary name contained in the target field is the same as the second dictionary name in the existing dictionary table, but the first dictionary value corresponding to the first dictionary name in the target field is different from the second dictionary value corresponding to the second dictionary name in the existing dictionary table, modify the second dictionary value in the existing dictionary table to the first dictionary value.

[0232] In one example, before modifying the second dictionary value in the existing dictionary table to the first dictionary value, the method further includes:

[0233] Display the prompt information for modifying the dictionary value and the information to confirm whether to modify it;

[0234] Modifying the second dictionary value in the existing dictionary table to the first dictionary value includes:

[0235] If a message confirming the modification is received, the second dictionary value in the existing dictionary table is modified to the first dictionary value.

[0236] For the device and equipment embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0237] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0238] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.

[0239] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0240] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A code field recognition method, characterized in that: The method comprises: Count the values ​​of the target fields in the data table to obtain statistical data; If the statistical data meets the preset condition, the characteristic value of the preset field characteristic corresponding to the target field is obtained according to the statistical data; wherein the statistical data includes the number of statistical records, the vacancy rate, and the code distribution, and the preset field characteristic includes the number of statistical records of the field, the number of code types, the maximum and minimum values ​​of the number of occurrences of each code, and the actual number of records; the actual number of records is equal to the product of the number of statistical records and the first ratio, and the first ratio is the ratio of the actual total number of records of the target field to the number of records sampled in the statistics; the preset condition is that the vacancy rate of the value of the target field in the statistical data is less than the preset vacancy rate threshold, and the code distribution data of the target field in the statistical data is not empty; The feature value is input into a trained code field recognition model, and whether the target field is a code field is determined according to the output result of the code field recognition model; wherein the code field is a field whose stored value is a numerical value or code in a limited range.

2. The method according to claim 1, characterized in that Obtaining a feature value of a preset field feature corresponding to the target field according to the statistical data includes: Cleaning the statistical data to obtain target data; According to the target data, a feature value of a preset field feature corresponding to the target field is determined.

3. The method according to claim 1, characterized in that The process of obtaining the code field identification model includes: Setting up the machine learning model; Acquire sample data, where the sample data includes a feature value of a preset field feature corresponding to a sample field and a label value corresponding to the sample field, where the label value is used to indicate whether the sample field is a code field; The machine learning model is trained using the sample data to obtain a trained machine learning model, and the trained machine learning model is used as a code field recognition model.

4. The method according to claim 1, characterized in that: The method further comprises: Obtain a target word vector corresponding to the target field; Inputting the feature value into a trained code field recognition model, and determining whether the target field is a code field according to an output result of the code field recognition model, including: The feature value and the target word vector are input into a trained code field recognition model, and whether the target field is a code field is determined according to an output result of the code field recognition model.

5. The method according to claim 1, characterized in that Also includes: If it is determined that the target field is a code field, a dictionary table is generated according to the dictionary name and the corresponding dictionary value contained in the target field.

6. The method according to claim 1, characterized in that Also includes: If it is determined that the target field is a code field, the existing dictionary table is modified according to the dictionary name and the corresponding dictionary value contained in the target field.

7. The method according to claim 6, characterized in that Modifying an existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field includes: If the existing dictionary table contains the dictionary table name corresponding to the target field, and the target field contains a target dictionary name that is not contained in the existing dictionary table, the target dictionary name and the corresponding dictionary value are added to the existing dictionary table.

8. The method according to claim 6, characterized in that Modifying an existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field includes: If the existing dictionary table does not contain the dictionary table name corresponding to the target field, all dictionary names and corresponding dictionary values ​​contained in the target field are added to the existing dictionary table.

9. The method according to claim 6, characterized in that Modifying an existing dictionary table according to the dictionary name and the corresponding dictionary value contained in the target field includes: If the first dictionary name contained in the target field is the same as the second dictionary name in the existing dictionary table, but the first dictionary value corresponding to the first dictionary name in the target field is different from the second dictionary value corresponding to the second dictionary name in the existing dictionary table, modify the second dictionary value in the existing dictionary table to the first dictionary value.

10. The method according to claim 9, characterized in that Before modifying the second dictionary value in the existing dictionary table to the first dictionary value, the method further includes: Display the prompt information for modifying the dictionary value and the information to confirm whether to modify it; Modifying the second dictionary value in the existing dictionary table to the first dictionary value includes: If a message confirming the modification is received, the second dictionary value in the existing dictionary table is modified to the first dictionary value.

11. A code field recognition device, characterized in that: The device comprises: The statistics module is used to count the values ​​of the target fields in the data table and obtain statistical data; A feature value acquisition module is used to obtain a feature value of a preset field feature corresponding to the target field according to the statistical data if the statistical data meets a preset condition; wherein the statistical data includes the number of statistical records, the vacancy rate, and the code distribution, and the preset field feature includes the number of statistical records of the field, the number of code types, the maximum and minimum values ​​of the number of occurrences of each code, and the actual number of records; the actual number of records is equal to the product of the number of statistical records and a first ratio, and the first ratio is the ratio of the actual total number of records of the target field to the number of records sampled in the statistics; the preset condition is that the vacancy rate of the value of the target field in the statistical data is less than a preset vacancy rate threshold, and the code distribution data of the target field in the statistical data is not empty; A determination module is used to input the feature value into a trained code field recognition model, and determine whether the target field is a code field according to the output result of the code field recognition model; wherein the code field is a field whose stored value is a numerical value or code in a limited range.

12. An electronic device, characterized in that: comprising a processor and a memory for storing executable instructions of the processor; The processor is configured to: Count the values ​​of the target fields in the data table to obtain statistical data; If the statistical data meets the preset condition, the characteristic value of the preset field characteristic corresponding to the target field is obtained according to the statistical data; wherein the statistical data includes the number of statistical records, the vacancy rate, and the code distribution, and the preset field characteristic includes the number of statistical records of the field, the number of code types, the maximum and minimum values ​​of the number of occurrences of each code, and the actual number of records; the actual number of records is equal to the product of the number of statistical records and the first ratio, and the first ratio is the ratio of the actual total number of records of the target field to the number of records sampled in the statistics; the preset condition is that the vacancy rate of the value of the target field in the statistical data is less than the preset vacancy rate threshold, and the code distribution data of the target field in the statistical data is not empty; The feature value is input into a trained code field recognition model, and whether the target field is a code field is determined according to the output result of the code field recognition model; wherein the code field is a field whose stored value is a numerical value or code in a limited range.

13. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed, the following processing is performed: Count the values ​​of the target fields in the data table to obtain statistical data; If the statistical data meets the preset condition, the characteristic value of the preset field characteristic corresponding to the target field is obtained according to the statistical data; wherein the statistical data includes the number of statistical records, the vacancy rate, and the code distribution, and the preset field characteristic includes the number of statistical records of the field, the number of code types, the maximum and minimum values ​​of the number of occurrences of each code, and the actual number of records; the actual number of records is equal to the product of the number of statistical records and the first ratio, and the first ratio is the ratio of the actual total number of records of the target field to the number of records sampled in the statistics; the preset condition is that the vacancy rate of the value of the target field in the statistical data is less than the preset vacancy rate threshold, and the code distribution data of the target field in the statistical data is not empty; The feature value is input into a trained code field recognition model, and whether the target field is a code field is determined according to the output result of the code field recognition model; wherein the code field is a field whose stored value is a numerical value or code in a limited range.

Images