Artificial intelligence-based data standard extraction method, device, equipment and medium

Abstract

The application provides an artificial intelligence-based data standard extraction method and device, electronic equipment and storage medium. The artificial intelligence-based data standard extraction method comprises the following steps: collecting business data in a business database to obtain a business basic data set; extracting code value type fields in the business basic data set to obtain an enumeration value list; generating a field vector based on the code value type fields to obtain a plurality of types of field groups; calculating the similarity between the enumeration value list to construct a code value similarity matrix; constructing a connected graph based on the code value similarity matrix to obtain a plurality of field connected graphs; extracting code value information of the code value type fields, and fusing the code value information based on the field connected graphs to obtain a data standard of the business database. The application can comprehensively consider the code value information in the field annotation and the association relationship between the fields, and obtain the data standard by using a graph algorithm, so as to reduce the redundancy of the data standard and improve the use efficiency of the database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a data standard extraction method and device based on artificial intelligence, an electronic device and a storage medium. BACKGROUND

[0002] Data standard extraction is an important part of a big data management system. When a database reaches a certain size, with the increase in the number of tables and fields of each table and the increase in the number of database system participants, data standards become numerous and non-uniform, resulting in an increase in the use cost of the database system. Therefore, extracting uniform data standards is an important part of data management, which can reduce the use cost of the database system and improve the efficiency of database business, thereby achieving the effect of reducing cost and increasing efficiency.

[0003] In traditional data standard extraction, each field in the database is usually treated as an independent field, and the association relationship between the database fields is ignored. In fact, in a large database system, the association relationship within a table or even between tables is not uncommon. For example, the value of a field can usually be obtained from the joint query of multiple tables. This mapping relationship is called an association relationship. For fields with the same mapping relationship, their data standards are the same. Ignoring such an association relationship often leads to redundancy of data standards, thereby reducing the use efficiency of the database. SUMMARY

[0004] In view of the above, it is necessary to provide a data standard extraction method and device based on artificial intelligence, an electronic device and a storage medium, to solve the technical problem of how to reduce the redundant part of the data standard, thereby improving the use efficiency of the database.

[0005] The present application provides a data standard extraction method based on artificial intelligence, which comprises:

[0006] Collecting business data in a business database to obtain a business basic data set;

[0007] Extracting code value type fields in the business basic data set to obtain an enumeration value list, the code value type fields corresponding one-to-one to the enumeration value list;

[0008] Generating a field vector based on the code value type fields, and grouping the code value type fields based on the field vector to obtain a plurality of categories of field groups;

[0009] Calculating the similarity between the enumeration value lists corresponding to the code value type fields in the field groups to construct a code value similarity matrix;

[0010] Constructing a connected graph between the code value type fields based on the code value similarity matrix to obtain a plurality of field connected graphs;

[0011] extracting code value information of the code value type field, and fusing the code value information based on the field connection graph to obtain data criteria of the service database.

[0012] In some embodiments, the collecting service data in the service database to obtain the service base data set comprises:

[0013] accessing system tables and system views of the service database to collect metadata of the service database;

[0014] generating a service data query statement according to a preset service data sampling ratio;

[0015] collecting service data in the service database based on the metadata and the service data query statement to obtain the service base data set.

[0016] In some embodiments, the extracting code value type fields in the service base data set to obtain an enumeration value list, the code value type fields corresponding to the enumeration value list one-to-one, comprises:

[0017] removing empty fields in the service base data set to obtain a service field data set;

[0018] extracting repeated fields in the service field data set as code value type fields;

[0019] enumerating specific values of the code value type fields as enumeration values, and counting frequencies of the enumeration values appearing in the service base data set;

[0020] constructing an enumeration value list corresponding to each code value type field based on the enumeration values and the frequencies.

[0021] In some embodiments, the generating a field vector based on the code value type fields, and grouping the code value type fields based on the field vector to obtain a plurality of categories of field groups, comprises:

[0022] converting the code value type fields into field vectors according to a word vector model;

[0023] calculating similarities between the field vectors;

[0024] dividing the code value type fields into a plurality of categories of field groups based on the similarities and a preset similarity threshold.

[0025] In some embodiments, the enumeration value list is a key-value pair structure, and the calculating similarities between the enumeration value lists corresponding to the code value type fields in the field groups to construct a code value similarity matrix, comprises:

[0026] converting keys in the enumeration value lists corresponding to the code value type fields in the field groups into key vectors.

[0027] calculate the similarity between each key vector to construct a key similarity matrix;

[0028] take the values in the enumeration value list corresponding to each code value type field in the field group as a value vector;

[0029] calculate the similarity between each value vector to construct a value similarity matrix;

[0030] perform weighted summation on the key similarity matrix and the value similarity matrix to obtain a code value similarity matrix.

[0031] In some embodiments, constructing a connected graph between each code value type field based on the code value similarity matrix to obtain a plurality of field connected graphs comprises:

[0032] obtain a comparison result by comparing each code value similarity in the code value similarity matrix with a preset code value similarity threshold;

[0033] update the code value similarity matrix based on the comparison result to obtain a graph companion matrix;

[0034] construct a connected graph between each code value type field based on the graph companion matrix to obtain a plurality of field connected graphs.

[0035] In some embodiments, the code value information of the code value type field is extracted, and the code value information is fused based on the field connected graph to obtain the data standard of the business database, comprising:

[0036] extract code value information from the annotations of the code value type field according to a preset language rule;

[0037] merge the code value type fields represented by the vertices of each field connected graph into the same group to obtain a code value type field group;

[0038] merge the code value information of each code value type field in the code value type field group as the data standard of the code value type field group;

[0039] take the data standard of all code value type field groups as the data standard of the business database.

[0040] The embodiments of the present application also provide a data standard extraction device based on artificial intelligence, comprising a collection module, an extraction module, a grouping module, a calculation module, a construction module and a fusion module:

[0041] The collection module is used to collect business data in a business database to obtain a business basic data set;

[0042] The extraction module is configured to extract code value type fields in the business basic data set to obtain an enumeration value list, the code value type fields being in one-to-one correspondence with the enumeration value list.

[0043] The grouping module is configured to generate a field vector based on the code value type fields and group the code value type fields based on the field vector to obtain a plurality of types of field groups.

[0044] The calculation module is configured to calculate the similarity between enumeration value lists corresponding to code value type fields in the field groups to construct a code value similarity matrix.

[0045] The construction module is configured to construct a connected graph between code value type fields based on the code value similarity matrix to obtain a plurality of field connected graphs.

[0046] The fusion module is configured to extract code value information of the code value type fields and fuse the code value information based on the field connected graphs to obtain a data standard of the business database.

[0047] Embodiments of the present application also provide an electronic device, which comprises:

[0048] a memory configured to store at least one instruction;

[0049] a processor configured to execute the instruction stored in the memory to implement the data standard extraction method based on artificial intelligence.

[0050] Embodiments of the present application also provide a computer readable storage medium, which stores at least one instruction, the at least one instruction being executed by a processor in an electronic device to implement the data standard extraction method based on artificial intelligence.

[0051] The present application comprehensively considers code value information in field annotations and the association between fields by constructing a code value similarity matrix based on collected business data, and obtains complete data standards in combination with a graph algorithm, thereby reducing the redundancy of data standards, improving the use efficiency of databases, and reducing the operation cost of databases. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flowchart of a preferred embodiment of the data standard extraction method based on artificial intelligence involved in the present application.

[0053] Figure 2 is a functional module diagram of a preferred embodiment of the data standard extraction apparatus based on artificial intelligence involved in the present application.

[0054] Figure 3is a structural schematic diagram of an electronic device of a preferred embodiment of the artificial intelligence-based data standard extraction method involved in the present application.

[0055] Figure 4 is a structural example diagram of a code value similarity matrix involved in the present application.

[0056] Figure 5 is a structural example diagram of a graph incidence matrix involved in the present application.

[0057] Figure 6 is a structural example diagram of a field connectivity graph involved in the present application.

[0058] Figure 7 is a basic information example diagram of a code value type field involved in the present application.

[0059] Figure 8 is a structural example diagram of an extracted data standard involved in the present application. DETAILED DESCRIPTION

[0060] In order to more clearly understand the objects, features and advantages of the present application, the present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application, and the described embodiments are only some of the embodiments of the present application, but not all the embodiments.

[0061] In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0063] The embodiment of the present application provides a data standard extraction method based on artificial intelligence, which can be applied to one or more electronic devices. The electronic device is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. The hardware of the electronic device includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0064] The electronic device can be any electronic product capable of human-computer interaction with a customer, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive Internet Protocol Television (IPTV), a smart wearable device, etc.

[0065] The electronic device can further include a network device and / or a customer device. The network device includes but is not limited to a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing.

[0066] The network in which the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0067] As shown in FIG. 1, it is a flowchart of a preferred embodiment of the data standard extraction method based on artificial intelligence of the present application. According to different needs, the order of steps in the flowchart can be changed, and some steps can be omitted. Figure 1

[0068] S10, collecting service data in a service database to obtain a service basic data set.

[0069] In an optional embodiment, the collecting service data in a service database to obtain a service basic data set includes:

[0070] Accessing a system table and a system view of the service database to collect metadata of the service database;

[0071] Generating a service data query statement according to a preset service data sampling ratio;

[0072] ​Based on the metadata and the business data query statement, business data in the business database is collected to obtain a business basic data set.

[0073] In this optional embodiment, the metadata in the business database can be collected by logging into the database access system and accessing the system table and system view of the business database through system commands. The business database can be a database supporting SQL statements such as MySQL, SQL Server, Oracle, Sybase, etc. The metadata is used to describe the attribute information of the business data, such as database table name, field name, field annotation, data type, constraint condition, table relationship, etc., to support functions such as indicating storage location, historical data, resource search, file record, etc.

[0074] In this optional embodiment, a suitable business data sampling rate can be selected to generate a business data query statement. The business data query statement collects a corresponding amount of business data from the storage location indicated by the metadata according to the business data sampling rate, thereby realizing the sampling collection of all business data in the business database. In this scheme, the business data sampling rate is set to 10%, that is, 10% of the business data in each table of the business database is randomly selected as the sampling data, and all the sampling data is taken as the business basic data set.

[0075] For example, taking the MySQL database as an example, the generated business data query statement is:

[0076] SELECT*FROM table name ORDER By rand()LIMIT sampling data, wherein the sampling data of each table is equal to the product of the total amount of business data of each table and the business data sampling rate.

[0077] In this way, a certain amount of business data can be randomly collected from each table of the business database to form a business basic data set, which provides data support for extracting data standards in the subsequent process.

[0078] S11, extracting code value type fields in the business basic data set to obtain an enumeration value list, the code value type fields corresponding to the enumeration value list one by one.

[0079] In an optional embodiment, the extracting code value type fields in the business basic data set to obtain an enumeration value list, the code value type fields corresponding to the enumeration value list one by one, comprises:

[0080] Removing empty fields in the business basic data set to obtain a business field data set;

[0081] Extracting repeated fields in the business field data set as code value type fields;

[0082] Enumerate the specific values ​​of the code value class field as enumeration values, and count the frequency of the enumeration values ​​in the business basic dataset;

[0083] Based on the enumerated values ​​and the frequency, an enumerated value list corresponding to each code value class field is constructed.

[0084] In this optional embodiment, unnecessary empty fields in the business base dataset can be removed by filling in null or empty values ​​in the SQL statement, and the business base dataset that does not contain empty fields can be used as the business field dataset.

[0085] In this optional embodiment, the repeated fields in the business field dataset can be extracted as code value fields by using the SQL statement "select * from table_name where condition", and then enumerated, that is, the specific values ​​of the code value fields are listed one by one as enumeration values. At the same time, the frequency of the enumeration values ​​in the business basic dataset is recorded, thereby constructing an enumeration value list corresponding to each code value field, wherein the enumeration value list is a key-value pair structure.

[0086] For example, the enumeration list of code value field A is: {'ID card':917,'driver's license':83}, indicating that code value field A only has two keys, ID card and driver's license, in the business basic dataset. ID card appears 917 times and driver's license appears 83 times. 'ID card':917 is one key-value pair and 'driver's license':83 is another key-value pair.

[0087] In this way, data redundancy can be reduced by removing empty fields, and at the same time, constructing an enumeration value list can extract the valid information of each code value class field in the form of key-value pairs, which facilitates the improvement of the efficiency of subsequent processes in calculating data standards.

[0088] S12, generate a field vector based on the code value class field, and group the code value class field based on the field vector to obtain field groups of multiple categories.

[0089] In an optional embodiment, the step of generating a field vector based on the code value class field, and grouping the code value class field into multiple categories of field groups based on the field vector, includes:

[0090] Based on the word vector model, the code value class fields are converted into field vectors;

[0091] Calculate the similarity between the vectors of each field;

[0092] Based on the similarity and a preset similarity threshold, the code value field is divided into multiple categories of field groups.

[0093] In an optional embodiment, the Chinese name of the code value class field can be converted into a field vector through a word vector model, which can be one of word2vec, glove, ELMo, BERT, etc., without specific limitation in the present solution.

[0094] In the optional embodiment, the similarity between the field vectors can be calculated through a cosine similarity algorithm to measure the association relationship between the code value class fields, and the value range of the similarity is [0, 1], and the code value class fields are divided into various categories of field groups through a preset similarity threshold. The similarity threshold can be 0.7, if the similarity between two field vectors is not less than 0.7, the code value class fields corresponding to the two field vectors are classified into the same category of field group, if the similarity between two field vectors is less than 0.7, the code value class fields corresponding to the two field vectors need to be classified into different categories of field groups.

[0095] In the optional embodiment, if a category of field group includes multiple code value class fields, and the cosine similarity between the field vector of any code value class field and the field vector of the code value class field to be divided is not less than 0.7, the code value class field to be divided can be classified into the category of field group.

[0096] For example, the “type of certificate accepted”, “type of certificate of buyer”, “type of certificate of employee”, and “type of certificate of group” can be classified into the same category of field group representing the type of certificate through the Chinese name of the code value class field.

[0097] In this way, the code value class fields with an association relationship can be classified into the same field group by calculating the similarity between the code value class fields, which facilitates the extraction of accurate data standards from the same category of field group in the subsequent process.

[0098] S13, calculating the similarity between the enumeration value lists corresponding to the code value class fields in the field group to construct a code value similarity matrix.

[0099] In an optional embodiment, the calculation of the similarity between the enumeration value lists corresponding to the code value class fields in the field group to construct a code value similarity matrix includes:

[0100] Converting the keys in the enumeration value lists corresponding to the code value class fields in the field group into key vectors;

[0101] Calculating the similarity between the key vectors to construct a key similarity matrix;

[0102] Taking the values in the enumeration value lists corresponding to the code value class fields in the field group as value vectors;

[0103] similarity between each value vector is calculated to construct a value similarity matrix;

[0104] The key similarity matrix and the value similarity matrix are weighted summed to obtain a code value similarity matrix.

[0105] In an optional embodiment, the key values in the enumeration value list corresponding to each code value class field in the field group are converted into a key vector composed of numerical codes by using one-hot encoding, and the Jaccard similarity between each key vector obtained is calculated by using the Jaccard similarity algorithm, and a key similarity matrix is constructed according to the Jaccard similarity and the key values. In this scheme, the Jaccard similarity between the key vectors of each code value class field is taken as the first similarity between the corresponding code value class fields, wherein any element C ij is used to represent the first similarity between the i th code value class field and the j th code value class field in the field group.

[0106] The one-hot encoding is also called one-bit effective encoding, which uses an N-bit state register to encode N states, each state has its own register bit, and is expressed in binary form. For example, assuming that the gender of a student is encoded as

male, female

[0107] Male -> [1, 0]; Female -> [0, 1].

[0108] In this optional embodiment, the Jaccard similarity algorithm is mainly used to calculate the similarity between sample sets. Since each enumeration value list includes multiple key values, the Jaccard coefficient can be calculated by calculating the ratio between the number of elements in the intersection of the key value sets of two enumeration value lists and the number of elements in the union, which reflects the Jaccard similarity between the two. The value range of the Jaccard coefficient is [0, 1], and the closer to 1, the higher the similarity between the two.

[0109] In this optional embodiment, since the enumeration values are in numerical form, the values in the enumeration value list corresponding to each code value class field in the field group can be directly taken as a value vector, and the cosine similarity between each value vector is measured by using the cosine similarity algorithm, thereby constructing a value similarity matrix. In this scheme, the similarity between the value vectors of each code value class field is taken as the second similarity between the corresponding code value class fields, wherein any element R ij is used to represent the second similarity between the i th code value class field and the j th code value class field in the field group.

[0110] In this optional embodiment, different weights can be assigned to the key similarity matrix and the value similarity matrix, so that the combined first and second similarities are obtained as the code value similarity through a weighted summation of corresponding elements of the two matrices. In this scheme, the weights assigned to the key similarity matrix and the value similarity matrix can be 0.6 and 0.4, respectively. That is, K ij =0.6C ij +0.4R ij , which represents the code value similarity between the i-th code value class field and the j-th code value class field in the field group.

[0111] For example, such as Figure 4 As shown, this is a group of fields representing document types. Fields 1 to 4 represent the code value fields "Acceptor's Document Type", "Buyer's Document Type", "Employee's Document Type", and "Group Document Type", respectively. The first similarity between field 1 and fields 2, 3, and 4 is 1, 0.9, and 0.4, respectively, and the second similarity is 0.5, 0.4, and 0.4, respectively. Substituting this into K... ij =0.6C ij +0.4R ij After calculation, the code value similarities between field 1 and fields 2, 3, and 4 are 0.8, 0.7, and 0.4, respectively. Similarly, the code value similarities between other fields can be calculated, thus forming the following... Figure 4 The code value similarity matrix shown is shown.

[0112] Thus, by comprehensively calculating the key similarity and value similarity among the enumeration value lists corresponding to each code value class field, a more accurate code value similarity can be obtained, thereby improving the accuracy of the data standard extracted in subsequent processes.

[0113] S14, Based on the code value similarity matrix, construct a connectivity graph between each code value class field to obtain multiple field connectivity graphs.

[0114] In an optional embodiment, the step of constructing a connectivity graph between each code value class field based on the code value similarity matrix to obtain a multi-field connectivity graph includes:

[0115] The comparison result is obtained by comparing the similarity of each code value in the code value similarity matrix with a preset code value similarity threshold;

[0116] The code value similarity matrix is ​​updated based on the comparison results to obtain the graph adjoint matrix;

[0117] Based on the graph adjoint matrix, a connectivity graph between each code value class field is constructed to obtain multiple field connectivity graphs.

[0118] In the optional embodiment, a suitable code value similarity threshold value can be selected. If each code value similarity in the code value similarity matrix is not less than the code value similarity threshold value, it indicates that the corresponding two code value class fields are highly similar. Therefore, a comparison result can be obtained by comparing each code value similarity in the code value similarity matrix with the preset code value similarity threshold value. The comparison result includes a code value similarity less than the code value similarity threshold value and a code value similarity not less than the code value similarity threshold value.

[0119] In the optional embodiment, as shown in Figure 5 If the code value similarity is less than the code value similarity threshold value, it is considered that the corresponding two code value class fields are irrelevant. The code value similarity is set to 0. If the code value similarity is not less than the code value similarity threshold value, the corresponding code value similarity is set to 1. Thus, all element values in the code value similarity matrix are replaced and updated. In the present scheme, the code value similarity matrix with all updated element values is used as a graph incidence matrix. The code value similarity threshold value can be 0.85.

[0120] In the optional embodiment, two code value class fields with a corresponding position of 1 in the graph incidence matrix are used as two connectable vertices. Two code value class fields with a corresponding position of 0 have no connection relationship, thereby constructing a field connection graph between each code value class field.

[0121] For example, as shown in the graph incidence matrix in Figure 5 The element value in the second row and the third column is 1, indicating that the corresponding code value class field 2 and the code value class field 3 are connectable. At the same time, the element value in the third row and the fourth column is 0, indicating that the corresponding code value class field 3 and the code value class field 4 have no connection relationship. Thus, the field connection graph as shown in Figure 6 is obtained.

[0122] In this way, the field connection graph between each code value class field can be generated by the generated graph incidence matrix. The association relationship between each code value class field is clearly expressed through the graph.

[0123] In S15, code value information of the code value class field is extracted, and the code value information is fused based on the field connection graph to obtain a data standard of the service database.

[0124] In an optional embodiment, the extraction of the code value information of the code value class field and the fusion of the code value information based on the field connection graph to obtain the data standard of the service database include:

[0125] Code value information is extracted from the annotation of the code value class field according to a preset language rule;

[0126] The code value class fields represented by the vertices of each field connectivity graph are merged into a same group to obtain a code value class field group;

[0127] The code value information of each code value class field in the code value class field group is merged to obtain data criteria of the code value class field group;

[0128] The data criteria of all code value class field groups are taken as data criteria of the business database.

[0129] In the optional embodiment, the code value information can be extracted from the annotations of the code value class fields by setting language rules. For example, if the Chinese annotation of code value class field B is "purchase channel 0: external purchase, 1: internal purchase", the code value information that can be extracted is "0: external purchase, 1: internal purchase". Generally, special symbols such as ":" and "-" are used to connect the code value information in the annotations of the code value class fields. The code value information can be extracted by pre-writing language rules and combining regular expressions to generate matching scripts.

[0130] For example, if the Chinese annotation of code value class field C is "certificate type 01: ID card, 02: passport", the code value information that can be extracted is "01: ID card, 02: passport".

[0131] In the optional embodiment, for each field connectivity graph, the code value class fields corresponding to the vertices connected in the field connectivity graph are merged into a same group, and the group obtained after the merging is taken as a code value class field group. Then the code value information of each code value class field in the code value class field group is merged to obtain data criteria of the code value class field group. In this scheme, the data criteria of all code value class field groups are finally taken as data criteria common to the business database.

[0132] For example, as shown in the code value class fields in Figure 7 , the data criteria obtained after the code value information is finally merged is as shown in Figure 8 .

[0133] In this way, the complete code value information can be obtained by merging the code value information, so that more complete data criteria common to the database can be obtained.

[0134] Please refer to Figure 2 , Figure 2is a functional module diagram of a preferred embodiment of the data standard extraction device based on artificial intelligence of the present application. The data standard extraction device based on artificial intelligence 11 comprises an acquisition module 110, an extraction module 111, a grouping module 112, a calculation module 113, a construction module 114, and a fusion module 115. The unit / module referred to in the present application refers to a series of computer readable instruction segments capable of being executed by the processor 13 and capable of completing a fixed function, which is stored in the memory 12. In the present embodiment, the functions of each unit / module will be described in detail in subsequent embodiments.

[0135] In an optional embodiment, the acquisition module 110 is configured to acquire service data in a service database to obtain a service basic data set.

[0136] In an optional embodiment, the acquisition of the service data in the service database to obtain the service basic data set comprises:

[0137] Accessing system tables and system views of the service database to acquire metadata of the service database;

[0138] Generating a service data query statement according to a preset service data sampling ratio;

[0139] Acquiring service data in the service database based on the metadata and the service data query statement to obtain the service basic data set.

[0140] In an optional embodiment, the extraction module 111 is configured to extract code value type fields in the service basic data set to obtain an enumeration value list, the code value type fields being in one-to-one correspondence with the enumeration value list.

[0141] In an optional embodiment, the extraction of the code value type fields in the service basic data set to obtain the enumeration value list, the code value type fields being in one-to-one correspondence with the enumeration value list, comprises:

[0142] Removing empty fields in the service basic data set to obtain a service field data set;

[0143] Extracting repeated fields in the service field data set as code value type fields;

[0144] Enumerating specific values of the code value type fields as enumeration values and counting frequencies of the enumeration values in the service basic data set;

[0145] Constructing an enumeration value list corresponding to each code value type field based on the enumeration values and the frequencies.

[0146] In an optional embodiment, the grouping module 112 is configured to generate field vectors based on the code value category fields, and group the code value category fields based on the field vectors to obtain a plurality of category field groups.

[0147] In an optional embodiment, the generating field vectors based on the code value category fields, and grouping the code value category fields based on the field vectors to obtain a plurality of category field groups comprises:

[0148] converting the code value category fields into field vectors according to a word vector model;

[0149] calculating similarities between the field vectors;

[0150] dividing the code value category fields into a plurality of category field groups based on the similarities and a preset similarity threshold.

[0151] In an optional embodiment, the calculating module 113 is configured to calculate similarities between enumeration value lists corresponding to code value category fields in the field groups to construct a code value similarity matrix.

[0152] In an optional embodiment, the calculating similarities between enumeration value lists corresponding to code value category fields in the field groups to construct a code value similarity matrix comprises:

[0153] converting keys in the enumeration value lists corresponding to the code value category fields in the field groups into key vectors;

[0154] calculating similarities between the key vectors to construct a key similarity matrix;

[0155] taking values in the enumeration value lists corresponding to the code value category fields in the field groups as value vectors;

[0156] calculating similarities between the value vectors to construct a value similarity matrix;

[0157] performing weighted summation on the key similarity matrix and the value similarity matrix to obtain the code value similarity matrix.

[0158] In an optional embodiment, the keys in the enumeration value lists corresponding to the code value category fields in the field groups can be converted into key vectors composed of numerical codes by using one-hot encoding, and then the obtained key vectors can be used to calculate Jaccard similarities between the key vectors by using a Jaccard similarity algorithm, and a key similarity matrix can be constructed according to the Jaccard similarities and the keys. In this scheme, the Jaccard similarities between the key vectors of the code value category fields are taken as first similarities between the corresponding code value category fields, wherein any element C ij is used to represent a first similarity between an i-th code value category field and a j-th code value category field in the field groups.

[0159] The one-hot encoding is also called one-bit effective encoding, which uses an N-bit state register to encode N states, each state has its own register bit, and is expressed in binary form. For example, assuming that the gender of the student is encoded as

male, female

[0160] Male -> [1, 0]; Female -> [0, 1].

[0161] In this optional embodiment, the Jaccard similarity algorithm is mainly used to calculate the similarity between sample sets. Since each enumeration value list includes multiple key values, the Jaccard coefficient can be calculated by calculating the ratio of the number of elements in the intersection of the key value sets of the two enumeration value lists to the number of elements in the union, which reflects the Jaccard similarity between the two. The value range of the Jaccard coefficient is [0, 1], and the closer to 1, the higher the similarity between the two.

[0162] In this optional embodiment, since the enumeration value is in numerical form, the values in the enumeration value list corresponding to each code value class field in the field group can be directly used as value vectors, and the cosine similarity algorithm can be used to measure the cosine similarity between the value vectors, thereby constructing a value similarity matrix. In this scheme, the similarity between the value vectors of each code value class field is used as the second similarity between the corresponding code value class fields, wherein any element R ij is used to represent the second similarity between the i-th code value class field and the j-th code value class field in the field group.

[0163] In this optional embodiment, different weights can be assigned to the key similarity matrix and the value similarity matrix, so that the similarity of the comprehensive first similarity and the second similarity obtained by the weighted sum of the corresponding elements of the two matrices is used as the code value similarity. In this scheme, the weights assigned to the key similarity matrix and the value similarity matrix can be 0.6, 0.4. That is, K ij = 0.6C ij + 0.4R ij , which represents the code value similarity between the i-th code value class field and the j-th code value class field in the field group.

[0164] For example, as Figure 4As shown, it is a group of fields representing the type of certificate, field 1 to field 4 respectively represent code value type fields "accepting certificate type", "buyer certificate type", "employee certificate type" and "group certificate type", wherein the first similarity between field 1 and field 2, 3, 4 is 1, 0.9, 0.4, and the second similarity is 0.5, 0.4, 0.4 respectively, then K ij = 0.6C ij + 0.4R ij After calculation, the final code value similarity between field 1 and field 2, 3, 4 is 0.8, 0.7, 0.4 respectively, and the code value similarity between other fields can be calculated in the same way, thereby forming a code value similarity matrix as shown in Figure 4 .

[0165] In an optional embodiment, the construction module 114 is configured to construct a connected graph between each code value type field based on the code value similarity matrix to obtain a plurality of field connected graphs.

[0166] In an optional embodiment, the construction of a connected graph between each code value type field based on the code value similarity matrix to obtain a plurality of field connected graphs comprises:

[0167] Comparing each code value similarity in the code value similarity matrix with a preset code value similarity threshold to obtain a comparison result;

[0168] Updating the code value similarity matrix based on the comparison result to obtain a graph companion matrix;

[0169] Constructing a connected graph between each code value type field based on the graph companion matrix to obtain a plurality of field connected graphs.

[0170] In this optional embodiment, a suitable code value similarity threshold can be selected, if each code value similarity in the code value similarity matrix is not less than the code value similarity threshold, it means that the corresponding two code value type fields are highly similar, therefore, the comparison result can be obtained by comparing each code value similarity in the code value similarity matrix with a preset code value similarity threshold, the comparison result includes less than the code value similarity threshold and not less than the code value similarity threshold.

[0171] In this optional embodiment, as shown in Figure 5 , if the code value similarity is less than the code value similarity threshold, it is considered that the corresponding two code value type fields are irrelevant, and the code value similarity is set to 0, if the code value similarity is not less than the code value similarity threshold, the corresponding code value similarity is set to 1, thereby all element values in the code value similarity matrix are replaced and updated, and the code value similarity matrix with all element values updated is taken as the graph companion matrix in this scheme, wherein the code value similarity threshold can be 0.85.

[0172] In this optional embodiment, the two code value classes with corresponding positions of 1 in the graph adjoint matrix are regarded as two connected vertices, while the two code value classes with corresponding positions of 0 are not connected, thereby constructing a field connectivity graph between the various code value classes.

[0173] For example, such as Figure 5 In the accompanying matrix shown, the element in the second row and third column is 1, indicating that the corresponding code value class field 2 and code value class field 3 are connected; meanwhile, the element in the third row and fourth column is 0, indicating that the corresponding code value class field 3 and code value class field 4 are not connected, thus obtaining the following... Figure 6 The field connectivity graph shown.

[0174] In an optional embodiment, the fusion module 115 is used to extract the code value information of the code value class field, and fuse the code value information based on the field connectivity graph to obtain the data standard of the business database.

[0175] In an optional embodiment, the step of extracting the code value information of the code value class field and fusing the code value information based on the field connectivity graph to obtain the data standard of the business database includes:

[0176] Extract code value information from the annotations of the code value class field according to preset language rules;

[0177] The code value class fields represented by the vertices of each field's connected graph are merged into the same group to obtain the code value class field group;

[0178] The code value information of each code value field in the code value field group is merged as the data standard of the code value field group;

[0179] Use the data standard for all code value field groups as the data standard for the business database.

[0180] In this optional embodiment, code value information can be extracted from the annotations of the code value class field by setting language rules. For example, if the Chinese annotation of code value class field B is “Procurement Channel 0: External Procurement, 1: Internal Procurement”, the extracted code value information is “0: External Procurement, 1: Internal Procurement”. Generally, the code value information in the annotations of code value class fields will contain special symbols such as “:” and “-”. The code value information can be extracted by pre-writing language rules and combining them with regular expressions to generate a matching script.

[0181] For example, if the Chinese annotation of the code value field C is “document type 01: ID card, 02: passport”, the extracted code value information is “01: ID card, 02: passport”.

[0182] In the optional embodiment, for each field connectivity graph, the code value class fields corresponding to the vertices connected on the field connectivity graph can be merged into a same group, and the group obtained after the merging is taken as a code value class field group. Then the code value information of each code value class field in the code value class field group is merged to obtain data criteria of the code value class field group. In the scheme, the data criteria of all code value class field groups are finally taken as the data criteria of the service database.

[0183] As shown in the code value class field, Figure 7 the data criteria obtained after the code value information is finally merged is as shown in Figure 8 .

[0184] As can be seen from the above technical solutions, the present application can comprehensively consider the code value information in the field annotation and the association relationship between fields by constructing a code value similarity matrix using the collected service data, and obtain complete data criteria by combining a graph algorithm, thereby reducing the redundancy of the data criteria, improving the use efficiency of the database, and reducing the operation cost of the database.

[0185] Please refer to Figure 3 , which is a structural schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device 1 comprises a memory 12 and a processor 13. The memory 12 is configured to store computer readable instructions, and the processor 13 is configured to execute the computer readable instructions stored in the memory to implement the artificial intelligence-based data criteria extraction method described in any of the above embodiments.

[0186] In an optional embodiment, the electronic device 1 further comprises a bus, a computer program stored in the memory 12 and executable on the processor 13, such as an artificial intelligence-based data criteria extraction program.

[0187] Figure 3 Only the electronic device 1 with the memory 12 and the processor 13 is shown, and those skilled in the art can understand that Figure 3 the structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0188] In combination with Figure 1 , the memory 12 in the electronic device 1 stores a plurality of computer readable instructions to implement an artificial intelligence-based data criteria extraction method, and the processor 13 can execute the plurality of instructions to implement:

[0189] collecting service data in a service database to obtain a service base data set;

[0190] extracting code value type fields in the business base dataset to obtain an enumeration value list, the code value type fields corresponding to the enumeration value list one by one;

[0191] generating a field vector based on the code value type fields, and grouping the code value type fields based on the field vector to obtain a plurality of categories of field groups;

[0192] calculating the similarity between the enumeration value lists corresponding to the code value type fields in the field groups to construct a code value similarity matrix;

[0193] constructing a connected graph between the code value type fields based on the code value similarity matrix to obtain a plurality of field connected graphs;

[0194] extracting code value information of the code value type fields, and fusing the code value information based on the field connected graph to obtain a data standard of the business database.

[0195] Specifically, the specific implementation method of the processor 13 on the above instructions can refer to the description of the related steps in the corresponding embodiments, which will not be repeated here. Figure 1

[0196] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 can be a bus type structure or a star type structure. The electronic device 1 can also include more or less other hardware or software, or different component arrangements, for example, the electronic device 1 can also include an input / output device, a network access device, etc.

[0197] It should be noted that the electronic device 1 is only an example. Other existing or future electronic products, such as those adaptable to the present application, should also be included within the protection scope of the present application and are hereby incorporated by reference.

[0198] ​The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 12 can be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 12 can also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. The memory 12 can be used to store application software and various data installed in the electronic device 1, such as the code of the artificial intelligence-based data standard extraction program, and can also be used to temporarily store data that has been output or will be output.

[0199] The processor 13 can be composed of an integrated circuit in some embodiments, such as a single packaged integrated circuit or a plurality of packaged integrated circuits with the same or different functions, including one or more combinations of a central processing unit (CPU), a microprocessor, a digital processing chip, a graphics processor, and various control chips, etc. The processor 13 is the control unit of the electronic device 1, which connects various components of the entire electronic device 1 through various interfaces and lines, executes programs or modules stored in the memory 12 (such as the artificial intelligence-based data standard extraction program, etc.), and calls data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0200] The processor 13 executes the operating system and various application programs installed in the electronic device 1. The processor 13 executes the application programs to implement the steps in each of the above artificial intelligence-based data standard extraction method embodiments, such as Figure 1 the steps shown in the above embodiments.

[0201] The computer program can be divided into one or more units / modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more units / modules can be a series of computer-readable instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program can be divided into an acquisition module 110, an extraction module 111, a grouping module 112, a calculation module 113, a construction module 114, and a fusion module 115.

[0202] The integrated units in the form of software function modules described above can be stored in a computer-readable storage medium. The software function modules described above are stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute part of the artificial intelligence-based data standard extraction method described in the embodiments of the present application.

[0203] The integrated units / modules of the electronic device 1, if implemented in the form of software function modules and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiments can also be instructed by a computer program to complete related hardware devices, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by a processor.

[0204] The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memories, etc.

[0205] Further, the computer-readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc.; and the data storage area can store data created according to the use of the blockchain node, etc.

[0206] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. The blockchain is essentially a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block containing information of a batch of network transactions, used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0207] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one arrow is used in Figure 3 , but it does not mean that there is only one bus or one type of bus. The bus is arranged to realize the connection and communication between the memory 12, the at least one processor 13, etc.

[0208] The embodiment of the present application also provides a computer readable storage medium (not shown in the figure), which stores computer readable instructions. The computer readable instructions are executed by a processor in an electronic device to realize the artificial intelligence-based data standard extraction method described in any of the above embodiments.

[0209] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. In actual implementation, there can be another division manner.

[0210] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, i.e. they can be located in one place or distributed on multiple network modules. According to actual needs, some or all of the modules can be selected to achieve the purpose of the present embodiment.

[0211] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each module can exist physically, or two or more modules can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional modules.

[0212] Furthermore, the word "comprising" does not exclude other modules or steps, and the word "a" or "an" does not exclude a plurality. The data described can also be implemented as one single module or a single data, or implemented by software or hardware. The terms first, second, etc. are used to distinguish names, not to indicate any specific order.

[0213] Finally, it should be noted that the above embodiments are merely used to illustrate, but not to limit the technical solutions of the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalent replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A data standard extraction method based on artificial intelligence, characterized in that, The method includes: Collect business data from the business database to obtain the basic business dataset; Extract the code value class field from the business basic dataset to obtain an enumeration value list, wherein the code value class field corresponds one-to-one with the enumeration value list; A field vector is generated based on the code value class field, and the code value class field is grouped based on the field vector to obtain field groups of various categories; Calculate the similarity between the enumerated value lists corresponding to each code value class field in the field group to construct a code value similarity matrix; Based on the code value similarity matrix, a connectivity graph between each code value class field is constructed to obtain a multi-field connectivity graph; Extract the code value information of the code value class field, and fuse the code value information based on the field connectivity graph to obtain the data standard of the business database.

2. The data standard extraction method based on artificial intelligence as described in claim 1, characterized in that, The business data collected from the business database to obtain the basic business dataset includes: Access the system tables and system views of the business database to collect metadata from the business database; Generate business data query statements based on preset business data sampling ratios; Based on the metadata and the business data query statements, business data is collected from the business database to obtain the basic business dataset.

3. The data standard extraction method based on artificial intelligence as described in claim 1, characterized in that, The step of extracting code value class fields from the business basic dataset to obtain an enumeration value list, wherein each code value class field corresponds one-to-one with the enumeration value list, including: The business field dataset is obtained by removing empty fields from the business base dataset. Extract the duplicate fields from the business field dataset and use them as code value fields; Enumerate the specific values ​​of the code value class field as enumeration values, and count the frequency of the enumeration values ​​in the business basic dataset; Based on the enumerated values ​​and the frequency, an enumerated value list corresponding to each code value class field is constructed.

4. The data standard extraction method based on artificial intelligence as described in claim 1, characterized in that, The process of generating a field vector based on the code value class field, and grouping the code value class field into multiple categories of field groups based on the field vector, includes: Based on the word vector model, the code value class fields are converted into field vectors; Calculate the similarity between the vectors of each field; Based on the similarity and a preset similarity threshold, the code value field is divided into multiple categories of field groups.

5. The data standard extraction method based on artificial intelligence as described in claim 1, characterized in that, The enumerated value list is a key-value pair structure. The step of calculating the similarity between the enumerated value lists corresponding to each code value class field in the field group to construct a code value similarity matrix includes: Convert the keys in the enumeration value list corresponding to each code value class field in the field group into a key vector; Calculate the similarity between each key vector to construct a key similarity matrix; The values ​​in the enumeration list corresponding to each code value class field in the field group are used as a value vector; Calculate the similarity between each value vector to construct a value similarity matrix; The key similarity matrix and the value similarity matrix are weighted and summed to obtain the code-value similarity matrix.

6. The data standard extraction method based on artificial intelligence as described in claim 1, characterized in that, The step of constructing a connectivity graph between each code value class field based on the code value similarity matrix to obtain multiple field connectivity graphs includes: The comparison result is obtained by comparing the similarity of each code value in the code value similarity matrix with a preset code value similarity threshold; The code value similarity matrix is ​​updated based on the comparison results to obtain the graph adjoint matrix; Based on the graph adjoint matrix, a connectivity graph between each code value class field is constructed to obtain multiple field connectivity graphs.

7. The data standard extraction method based on artificial intelligence as described in claim 1, characterized in that, The step of extracting the code value information of the code value class field and fusing the code value information based on the field connectivity graph to obtain the data standard of the business database includes: Extract code value information from the annotations of the code value class field according to preset language rules; The code value class fields represented by the vertices of each field's connected graph are merged into the same group to obtain the code value class field group; The code value information of each code value field in the code value field group is merged to form the data standard of the code value field group; Use the data standard for all code value field groups as the data standard for the business database.

8. A data standard extraction device based on artificial intelligence, characterized in that, The device includes an acquisition module, an extraction module, a grouping module, a calculation module, a construction module, and a fusion module. The acquisition module is used to acquire business data from the business database to obtain the basic business dataset; The extraction module is used to extract code value class fields from the business basic dataset to obtain an enumeration value list, wherein the code value class fields correspond one-to-one with the enumeration value list; The grouping module is used to generate a field vector based on the code value class field, and to group the code value class field based on the field vector to obtain field groups of multiple categories; The calculation module is used to calculate the similarity between the enumeration value lists corresponding to each code value class field in the field group to construct a code value similarity matrix; The construction module is used to construct a connectivity graph between each code value class field based on the code value similarity matrix to obtain a multi-field connectivity graph; The fusion module is used to extract the code value information of the code value class field, and fuse the code value information based on the field connectivity graph to obtain the data standard of the business database.

9. An electronic device, characterized in that, The electronic device includes: Memory, which stores computer-readable instructions; and The processor executes computer-readable instructions stored in the memory to implement the artificial intelligence-based data standard extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the data standard extraction method based on artificial intelligence as described in any one of claims 1 to 7.

Images