Dictionary construction method and device, electronic equipment and medium

Abstract

The application discloses a dictionary construction method and device, electronic equipment and medium. Through the technical scheme of the application, the column label corresponding to the column data subjected to feature coding can be automatically recognized by using a preset character classification model, and the table label corresponding to the table data subjected to feature coding can be automatically recognized. Then, the obtained data and the corresponding label classification results are integrated by using a data dictionary generation tool to form a final data dictionary. Thus, on one hand, the purpose of intelligently constructing a data dictionary can be achieved, and the time-consuming and laborious drawbacks of manually establishing a data dictionary in the prior art are avoided. On the other hand, since the feature coding technology capable of completely extracting data semantic information is adopted, the data can be classified more carefully, and a more accurate data dictionary can be obtained.

Description

Technical Field

[0001] This application relates to data processing technology, and in particular to a method, apparatus, electronic device, and medium for constructing a dictionary. Background Technology

[0002] In the digital information age, data is a core asset for enterprises, and the advancement of digital transformation is inseparable from the in-depth mining and application of enterprise data. Currently, various enterprise platforms contain a large number of databases, each with tens of thousands of data tables. Therefore, meticulous and continuous maintenance of the data dictionaries in each database is an important guarantee for the daily data development and use of each organization.

[0003] In current technologies, enterprise information systems are often isolated, leading to problems such as inconsistent data standards and low transmission efficiency when transferring data between different databases. To address this issue, developers use data dictionaries to access and manage data tables within the respective systems. With the help of a data dictionary, data users can clearly understand the structure and business semantics of the data.

[0004] However, existing data dictionaries are mainly constructed manually, which leads to problems such as non-standard construction and low construction efficiency. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and medium for constructing a dictionary. These solutions address the problems of inconsistent construction methods and low efficiency inherent in manually constructed data dictionaries in related technologies.

[0006] According to one aspect of the embodiments of this application, a method for constructing a dictionary is provided, comprising:

[0007] Retrieve multiple column data sets and table data sets contained in the database, where the column data in each column data set belongs to the same column field label, and the table data in each table data set belongs to the same table topic label;

[0008] The feature encoding is performed on all column data contained in the multiple column data sets to obtain multiple column feature vectors; and the feature encoding is performed on all table data contained in the multiple table data sets to obtain multiple table feature vectors.

[0009] The multiple column feature vectors and the multiple table feature vectors are input into the pre-trained target label classification model to obtain the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector.

[0010] A data dictionary is constructed based on the column field labels and their corresponding column data, and based on the table topic labels and their corresponding table data.

[0011] Optionally, in another embodiment based on the method described above in this application, obtaining the multiple column data sets and table data sets contained in the database includes:

[0012] Retrieve all column field labels and all table topic labels contained in the database;

[0013] The actual semantics corresponding to each column field label are parsed separately, and the actual semantics corresponding to each table topic label are also parsed separately.

[0014] Column field labels with the same actual semantic meaning from multiple column field labels are merged to obtain filtered column field labels; and table topic labels with the same actual semantic meaning from multiple table topic labels are merged to obtain filtered table topic labels.

[0015] Extract the multiple column data sets from the filtered column field labels; and extract the multiple table data sets from the filtered table topic labels.

[0016] Optionally, in another embodiment based on the method described above in this application, the column data is feature-encoded according to a preset feature encoding rule to obtain the column feature vector; and the table data is feature-encoded according to the preset feature encoding rule to obtain the table feature vector;

[0017] The preset feature encoding rule includes at least one of the following encoding dimensions:

[0018] Dimensions of numerical proportion, time proportion, number of characters, proportion of specific characters, number of specific numbers, and number of specific characters and numbers.

[0019] Optionally, in another embodiment based on the method described above in this application, constructing a data dictionary based on the column field labels and corresponding column data, and based on the table topic labels and corresponding table data, includes:

[0020] Using a data dictionary generation tool, establish the first association between each column field label and its corresponding column data; and establish the second association between each table topic label and its corresponding table data.

[0021] According to the dictionary specifications recorded by the data dictionary generation tool, the first association relationship and the second association relationship are integrated to obtain the data dictionary to be improved;

[0022] The data dictionary to be improved is concatenated with the pre-built initial data dictionary to obtain the data dictionary.

[0023] Optionally, in another embodiment based on the method described above in this application, before concatenating the data dictionary to be improved with the pre-built initial data dictionary, the method further includes:

[0024] Retrieve the labels of multiple sample column fields and the topic labels of sample tables contained in the database;

[0025] Each sample column field label is associated with at least one sample column data, and each sample table topic label is associated with at least one sample table data;

[0026] According to the dictionary specifications recorded by the data dictionary generation tool, the sample column field labels and associated sample column data are integrated, and the sample table topic labels and associated sample table data are integrated to construct the initial data dictionary.

[0027] Optionally, in another embodiment based on the method described above in this application, after obtaining the multiple sample column field labels and sample table topic labels contained in the database, the method further includes:

[0028] The sample column field labels associated with the sample column data and the sample table topics associated with the sample table data are used as training data to train the initial label classification model until the target label classification model is obtained after training.

[0029] The target label classification model includes an input layer, a first hidden layer and a second hidden layer with ReLU activation function, and an output layer with softmax activation function.

[0030] Optionally, in another embodiment based on the method described above in this application, after constructing the data dictionary, the method further includes:

[0031] Obtain at least one other data dictionary, wherein the other data dictionary and the data dictionary are dictionaries with the same business data type;

[0032] Extract multiple other column data and other table data contained in the other data dictionary, and encode the other column data and other table data for features respectively to obtain multiple other column feature vectors and other table feature vectors;

[0033] Using the feature vectors of the other columns and the feature vectors of the other tables, other data dictionaries to be improved are obtained;

[0034] The data dictionary is improved by using the other data dictionaries that need to be improved.

[0035] According to another aspect of the embodiments of this application, a dictionary construction apparatus is provided, characterized in that it includes:

[0036] The acquisition module is configured to acquire multiple column data sets and table data sets contained in the database, wherein the column data in each column data set belongs to the same column field label, and the table data in each table data set belongs to the same table topic label;

[0037] The encoding module is configured to perform feature encoding on all column data contained in the multiple column data sets to obtain multiple column feature vectors; and to perform feature encoding on all table data contained in the multiple table data sets to obtain multiple table feature vectors.

[0038] The output module is configured to input the plurality of column feature vectors and the plurality of table feature vectors into the pre-trained target label classification model to obtain the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector.

[0039] The building module is configured to build a data dictionary based on the column field labels and their corresponding column data, and based on the table topic labels and their corresponding table data.

[0040] According to another aspect of the embodiments of this application, an electronic device is provided, comprising:

[0041] Memory, used to store executable instructions; and

[0042] A display is used in conjunction with the memory to execute the executable instructions to complete the dictionary construction method described above.

[0043] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided for storing computer-readable instructions, which, when executed, perform the operation of any of the dictionary construction methods described above.

[0044] In this embodiment, multiple column data sets and table data sets contained in a database can be obtained. Column data in each column data set belongs to the same column field label, and table data in each table data set belongs to the same table topic label. All column data in the multiple column data sets are feature-encoded to obtain multiple column feature vectors. Similarly, all table data in the multiple table data sets are feature-encoded to obtain multiple table feature vectors. These column feature vectors and table feature vectors are input into a pre-trained target label classification model to obtain the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector. A data dictionary is constructed based on the column field labels and their corresponding column data, and based on the table topic labels and their corresponding table data. By applying the technical solution of this application, a preset character classification model can automatically identify the column labels corresponding to feature-encoded column data and the table labels corresponding to feature-encoded table data. Then, a data dictionary generation tool is used to integrate the obtained data and the corresponding label classification results to form the final data dictionary. This approach achieves the goal of intelligently constructing a data dictionary, avoiding the time-consuming and labor-intensive drawbacks of manually building a data dictionary in existing technologies. Furthermore, because this application employs feature encoding technology capable of fully extracting semantic information from the data, it enables more detailed data classification, resulting in a more accurate data dictionary.

[0045] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0046] The accompanying drawings, which form part of this specification, illustrate embodiments of this application and, together with the description, serve to explain the principles of this application.

[0047] This application can be more clearly understood with reference to the accompanying drawings and the following detailed description, wherein:

[0048] Figure 1 A schematic diagram of a dictionary construction method provided in an embodiment of this application is shown;

[0049] Figure 2 A flowchart illustrating a dictionary construction method according to an embodiment of this application is shown;

[0050] Figure 3 This illustration shows a schematic diagram of feature encoding of data according to an embodiment of this application;

[0051] Figure 4 A schematic diagram of a preset feature encoding rule provided in an embodiment of this application is shown;

[0052] Figure 5A flowchart illustrating another dictionary construction method provided in an embodiment of this application is shown;

[0053] Figure 6 This illustration shows a diagram of information contained in a data dictionary provided in an embodiment of this application;

[0054] Figure 7 This invention provides a schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0055] Figure 8 This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0056] Figure 9 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0057] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0058] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0059] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the scope of this application or its application or use.

[0060] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0061] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0062] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0063] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0064] The following is combined Figures 1-6 This application describes a method for constructing a dictionary according to exemplary embodiments thereof. It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way. Rather, the embodiments of this application can be applied to any applicable scenario.

[0065] This application also proposes a method, apparatus, electronic device, and medium for constructing a dictionary.

[0066] Figure 1 A schematic flowchart illustrating a dictionary construction method according to an embodiment of this application is shown. Figure 1 As shown, the method includes:

[0067] S101, retrieve multiple column data sets and table data sets contained in the database, wherein the column data in each column data set belongs to the same column field label, and the table data in each table data set belongs to the same table topic label.

[0068] S102, perform feature encoding on all column data contained in multiple column data sets to obtain multiple column feature vectors; and perform feature encoding on all table data contained in multiple table data sets to obtain multiple table feature vectors.

[0069] S103, input multiple column feature vectors and multiple table feature vectors into the pre-trained target label classification model to obtain the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector.

[0070] S104. Construct a data dictionary based on column field labels and corresponding column data, and based on table topic labels and corresponding table data.

[0071] Furthermore, the construction of existing data dictionaries mainly follows these approaches:

[0072] Firstly, artificially constructed:

[0073] Table names and fields are often customized by developers based on business scenarios, personal experience and habits, and English proficiency (table names are usually in English). This leads to inconsistent naming conventions for table fields. Fields with the same business meaning may have multiple names, or fields with different business meanings may have the same name. Furthermore, inconsistent data dictionary management makes it easy for data dictionaries to be modified arbitrarily, resulting in multiple versions of the data dictionary, and even the latest version is inaccurate. In actual project development, tasks are often carried out in batches, corresponding to different data dictionaries maintained by different personnel, or new data dictionaries are created. Finally, maintenance personnel compare the differences between the various versions of the data dictionary to form a unified data dictionary. This process is labor-intensive, inefficient, and lacks unified standards, making it difficult to execute comprehensively and objectively, thus affecting project progress.

[0074] Secondly, the column names are constructed by word segmentation:

[0075] This method involves selecting words with nouns as their part of speech, labeling them, and then performing similarity matching on these nouns. Words with similarity scores above a certain threshold are considered to have the same meaning. While this method can indeed improve the accuracy of data dictionary construction, it requires a large number of labeled samples and is only suitable for cases where field names are in Chinese. However, many databases in various organizations contain numerous field names that are English abbreviations or arbitrarily chosen, making this method unsuitable in such cases.

[0076] Third, construct it using the table's metadata:

[0077] The method involves vectorizing metadata using one-hot encoding, then classifying the vectorized metadata using a document topic generation model, ultimately classifying field names and table topics. This method is highly efficient, determining the table's topic category using only its metadata without needing to read the entire table, offering advantages such as small data volume and fast processing speed. However, it is only suitable for situations where metadata exists and is relatively clear. If metadata is missing, this method is ineffective; if the amount of metadata information is limited, the accuracy will be very low.

[0078] In summary, existing methods for constructing data dictionaries have the following problems:

[0079] 1. Relying on manual methods is inefficient, and the versions and standards are not objective and inconsistent;

[0080] 2. Semantic methods have limited applicability and cannot be quickly reused across different scenarios;

[0081] 3. Non-semantic methods suffer from a severe lack of business semantic information;

[0082] 4. The required sample data is too large, resulting in high labor costs.

[0083] Furthermore, to address the aforementioned issues, this application proposes a dictionary construction method. The idea is to automatically identify the column labels corresponding to feature-encoded column data and the table labels corresponding to feature-encoded table data using a pre-defined character classification model. Then, a data dictionary generation tool is used to integrate the obtained data and the corresponding label classification results to form the final data dictionary.

[0084] For example, Figure 2 The diagram shown is a flowchart of a dictionary construction architecture proposed in this application. Specifically, it includes the following steps:

[0085] Step 1: Obtain the labels of multiple sample column fields and the topic labels of the sample table contained in the database.

[0086] Each sample column field label is associated with at least one sample column data, and each sample table topic label is associated with at least one sample table data.

[0087] Furthermore, column field labels are the content corresponding to the beginning of a column in a data table. In one understanding, a column field label can be seen as a categorization of the nature of the expanded column content. For example, list items containing attributes, age, name, education, employee ID, performance, etc., all belong to column field labels.

[0088] As an example, column data refers to the individual data recorded under the column field label. For instance, for a column field label that records the age, the column field label is "Age," and the age data of all users under the column field label "Age" (e.g., 18, 45, 33) is the column data associated with that column field label (Age).

[0089] To give another example, for the column field label of the record name table item, the column field label is "Name", and the name data of all users under the column field label "Name" (e.g., Zhang San, Li Si, Wang Wu) is the column data associated with the column field label (Name).

[0090] Furthermore, table topic tags are the content corresponding to the name of a specific table in a database. In one understanding, a table topic tag can be seen as a classification of the nature of its expanded table content. Examples include personnel information tables, salary tables, performance tables, and overdue tables. These all fall under the category of table topic tags.

[0091] As an example, table data refers to the individual data (including column data and column field labels) recorded under the table topic label. For instance, for the table topic label "Personnel Information Table," the table topic label is "Personnel Information Table," and all the data under this table topic label "Personnel Information Table," such as the column field label "Name" and the age data of all users associated with it (e.g., 18 years old, 45 years old, 33 years old), constitutes the table data associated with this table topic label (Personnel Information Table).

[0092] To give another example, for the table topic label of a table that records salary information, the table topic label is "Salary Table". All the data under the table topic label "Salary Table", such as the column field label "Name" and the age data of all users associated with it (e.g., 18 years old, 45 years old, 33 years old), and the column field label "Salary" and the salary data of all users associated with it (e.g., 10,000 yuan, 5,000 yuan, 3,500 yuan), are the table data associated with the table topic label (Salary Table).

[0093] In one approach, a data table may have one column field label or multiple column field labels. This application does not impose any limitations on this.

[0094] Step 2: Following the dictionary specifications recorded by the data dictionary generation tool, integrate the sample column field labels with the associated sample column data, and integrate the sample table topic labels with the associated sample table data to construct the initial data dictionary.

[0095] Furthermore, this application can obtain existing data dictionary generation tools and, based on the dictionary generation specifications recorded therein, integrate the data obtained in step 1 (i.e., sample column field labels and associated sample column data, and sample table topic labels and associated sample table data) according to the corresponding specifications, thereby obtaining an initial data dictionary.

[0096] Understandably, this initial data dictionary can serve two purposes: firstly, as samples for training the initial neural network model; and secondly, it can be used to concatenate subsequently generated data dictionaries to obtain a more comprehensive set of data dictionaries.

[0097] Step 3: Use the sample column field labels associated with the sample column data and the sample table topics associated with the sample table data as training data to train the initial label classification model until the target label classification model is obtained after training.

[0098] In one approach, to fully utilize the feature information of the sample data, an embodiment of this application can construct an initial label classification model composed of a fully connected neural network.

[0099] As an example, the initial label classification model consists of four layers: an input layer (338 dimensions), a first hidden layer (128 dimensions), a second hidden layer (64 dimensions), and an output layer (N dimensions). The two hidden layers use the ReLU activation function, while the output layer uses the softmax activation function. Similarly, a fully connected neural network can be constructed for topic classification, the difference being that the output layer is M-dimensional.

[0100] Furthermore, in the training process of the initial label classification model in this embodiment, the cross-entropy loss, which is common to classification, can be used. The fastest optimizer, Adam, is selected, and the learning rate is set to a cosine annealing function. This is to prevent the learning rate from being too large when the loss is too small, which would lead to unstable results. The sample data is divided into an 80% training set and a 20% validation set. The multi-dimensional feature vectors of the columns and tables in the training set are input into the corresponding models. Training ends when the difference between the losses of two iterations is less than a specified threshold. Thus, the final target label classification model is obtained.

[0101] Step 4: Obtain all column field labels and all table topic labels contained in the database, and parse the actual semantics corresponding to each column field label and each table topic label respectively.

[0102] Step 5: Merge column field labels with the same actual semantic meaning from multiple column field labels to obtain filtered column field labels; and merge table topic labels with the same actual semantic meaning from multiple table topic labels to obtain filtered table topic labels.

[0103] Furthermore, since each database contains multiple tables, and each table may be created and maintained by different business personnel, situations may arise where column field labels and table subject labels have different names but the same underlying semantic information.

[0104] For example, regarding column field labels, although "age," "age number," and "age" are different expressions, their actual meanings are the same. Therefore, this application can merge column field labels with the same actual meaning to obtain filtered column field labels.

[0105] Similarly, for table topic tags, whether it's "Name Table," "Personnel Information Table," or "Employee Name Table," although they are different expressions, their actual meaning is the same. Therefore, this application can merge table topic tags with the same actual meaning from multiple table topic tags to obtain filtered table topic tags.

[0106] Step 6: Extract multiple column data sets from the filtered column field labels; and extract multiple table data sets from the filtered table topic labels.

[0107] Furthermore, after obtaining the filtered column field labels, this application can extract an unlimited number of column data from them. It should be noted that each column data belonging to the same column field label forms a column data set, thus resulting in multiple column data sets.

[0108] Furthermore, after obtaining the filtered table topic tags, this application can extract an unlimited number of table data from them. It should be noted that each set of table data belonging to the same table topic tag constitutes a table data collection. This results in multiple table data collections.

[0109] Step 7: Encode the column data according to the preset feature encoding rules to obtain the column feature vector; and encode the table data according to the preset feature encoding rules to obtain the table feature vector.

[0110] In one approach, such as Figure 3 As shown in the embodiments of this application, a feature encoding method is proposed to achieve feature encoding of data. This feature encoding method can completely extract the semantic information of the data and is applicable to most types of data. Specifically, it includes:

[0111] Step a: Convert the format of column data and table data to UTF-8 encoding;

[0112] Step b: Before each input model, randomly extract 1000 rows of data corresponding to each label, that is, randomly extract 1000 rows of column data corresponding to column labels and 1000 rows of table data corresponding to table topics, thereby obtaining the original column data and the original table data.

[0113] The original data for each column consists of 1000 rows and 1 column of data extracted for each column label. If one column label corresponds to multiple column fields, these column fields are first concatenated into one column field before extraction. The original data for each table consists of 1000 rows and N columns of data extracted for each topic label, where N represents the number of columns in the table. If one topic label corresponds to multiple tables, these multiple tables are first concatenated into one wide table before extraction. Null values ​​in the concatenated wide table are filled in a certain way.

[0114] Step c, perform feature encoding on the original column data and the original table data, the encoding rules are as follows: Figure 4 The features comprise a total of 11 major dimensions, which are further broken down into 338 minor dimensions:

[0115] The first major dimension includes the proportion of characters such as numbers to all characters, which constitutes three sub-dimensions;

[0116] The second major dimension includes the proportion of columns such as time / name to all columns, forming four sub-dimensions;

[0117] The third major dimension includes: the average number of characters per cell, which constitutes one sub-dimension;

[0118] The fourth major dimension includes the average number of characters, such as numbers, contained in each cell, which constitutes the three sub-dimensions.

[0119] The fifth dimension includes the standard deviation of the number of characters, such as numbers, contained in each cell, which constitutes the three sub-dimensions;

[0120] The sixth major dimension includes whether it contains characters such as 0, which constitutes 10 sub-dimensions;

[0121] The seventh dimension includes characters such as whether they are all 0s, which constitutes 10 sub-dimensions;

[0122] The eighth dimension includes statistical dimensions such as the mean number of each digit from 0 to 9, which constitute 70 sub-dimensions;

[0123] The ninth dimension includes whether it contains characters such as 'a', which constitutes 26 sub-dimensions;

[0124] The tenth dimension includes whether all characters are 'a', which constitutes 26 sub-dimensions;

[0125] The eleventh major dimension includes statistical dimensions such as the mean number of each letter in the letter a, z, which together constitute 182 minor dimensions.

[0126] All of the above subdivisions constitute a 338-dimensional feature vector. That is, after each sample data is feature-encoded, a 338-dimensional feature vector is formed, which serves as the input for the subsequent target label classification model.

[0127] Step 8: Input multiple column feature vectors and multiple table feature vectors into the pre-trained target label classification model to obtain the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector.

[0128] Furthermore, in this embodiment of the application, after obtaining the column feature vector and the table feature vector, they can be input into the target label classification model so that the target label classification model outputs the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector.

[0129] Step 9: Using a data dictionary generation tool, establish the first association between each column field label and its corresponding column data; and establish the second association between each table topic label and its corresponding table data.

[0130] Step 10: According to the dictionary specifications recorded by the data dictionary generation tool, integrate the first association relationship and the second association relationship to obtain the data dictionary to be improved.

[0131] Step 11: Combine the data dictionary to be improved with the pre-built initial data dictionary to obtain the data dictionary.

[0132] Furthermore, this application can also utilize existing data dictionary generation tools and, based on their recorded dictionary generation specifications, integrate the data obtained in the above steps (i.e., the first association relationship between column field labels and their respective column data, and the second association relationship between each table subject label and its respective table data) according to the corresponding specifications, thereby obtaining an initial data dictionary. This initial data dictionary is then concatenated with the existing data dictionary to form a more complete final data dictionary.

[0133] In this embodiment, multiple column data sets and table data sets contained in the database can be obtained, wherein the column data in each column data set belongs to the same column field label, and the table data in each table data set belongs to the same table topic label; feature encoding is performed on all column data contained in the multiple column data sets to obtain multiple column feature vectors; and feature encoding is performed on all table data contained in the multiple table data sets to obtain multiple table feature vectors; the multiple column feature vectors and multiple table feature vectors are input into a pre-trained target label classification model to obtain the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector; a data dictionary is constructed based on the column field labels and the corresponding column data, and based on the table topic labels and the corresponding table data.

[0134] By applying the technical solution of this application, a preset character classification model can be used to automatically identify the column labels corresponding to feature-encoded column data, and automatically identify the table labels corresponding to feature-encoded table data. Then, a data dictionary generation tool is used to integrate the obtained data and the corresponding label classification results to form the final data dictionary. This achieves the goal of intelligently constructing a data dictionary, avoiding the time-consuming and labor-intensive drawbacks of manually building data dictionaries in existing technologies. Furthermore, because this application uses feature encoding technology that can completely extract the semantic information of the data, it can classify the data more meticulously to obtain a more accurate data dictionary.

[0135] Optionally, in another embodiment based on the method described above in this application, obtaining multiple column data sets and table data sets contained in the database includes:

[0136] Retrieve all column field labels and all table topic labels contained in the database;

[0137] The actual semantics corresponding to each column field label and the actual semantics corresponding to each table topic label are analyzed separately.

[0138] Column field labels with the same actual semantic meaning from multiple column field labels are merged to obtain filtered column field labels; and table topic labels with the same actual semantic meaning from multiple table topic labels are merged to obtain filtered table topic labels.

[0139] Extract the multiple column data sets from the filtered column field labels; and extract the multiple table data sets from the filtered table topic labels.

[0140] Optionally, in another embodiment based on the method described above in this application, the column data is feature-encoded according to a preset feature encoding rule to obtain the column feature vector; and the table data is feature-encoded according to the preset feature encoding rule to obtain the table feature vector;

[0141] The preset feature encoding rule includes at least one of the following encoding dimensions:

[0142] Dimensions include: the proportion of numeric characters to all characters; the proportion of time / name columns to all columns; the average number of characters per cell; the average number of numeric characters per cell; the standard deviation of the number of numeric characters per cell; whether it contains zero characters; whether it consists entirely of zero characters; the mean number of each digit from 0 to 9; whether it contains the letter 'a'; whether it consists entirely of 'a' characters; and the mean number of each letter 'az'.

[0143] Optionally, in another embodiment based on the method described above in this application, constructing a data dictionary based on the column field labels and corresponding column data, and based on the table topic labels and corresponding table data, includes:

[0144] Using a data dictionary generation tool, establish the first association between each column field label and its corresponding column data; and establish the second association between each table topic label and its corresponding table data.

[0145] According to the dictionary specifications recorded by the data dictionary generation tool, the first association relationship and the second association relationship are integrated to obtain the data dictionary to be improved;

[0146] The data dictionary to be improved is concatenated with the pre-built initial data dictionary to obtain the data dictionary.

[0147] Optionally, in another embodiment based on the method described above in this application, before concatenating the data dictionary to be improved with the pre-built initial data dictionary, the method further includes:

[0148] Retrieve the labels of multiple sample column fields and the topic labels of sample tables contained in the database;

[0149] Each sample column field label is associated with at least one sample column data, and each sample table topic label is associated with at least one sample table data;

[0150] According to the dictionary specifications recorded by the data dictionary generation tool, the sample column field labels and associated sample column data are integrated, and the sample table topic labels and associated sample table data are integrated to construct the initial data dictionary.

[0151] Optionally, in another embodiment based on the method described above in this application, after obtaining the multiple sample column field labels and sample table topic labels contained in the database, the method further includes:

[0152] The sample column field labels associated with the sample column data and the sample table topics associated with the sample table data are used as training data to train the initial label classification model until the target label classification model is obtained after training.

[0153] The target label classification model includes an input layer, a first hidden layer and a second hidden layer with ReLU activation function, and an output layer with softmax activation function.

[0154] Optionally, in another embodiment based on the method described above in this application, after constructing the data dictionary, the method further includes:

[0155] Obtain at least one other data dictionary, wherein the other data dictionary and the data dictionary are dictionaries with the same business data type;

[0156] Extract multiple other column data and other table data contained in the other data dictionary, and encode the other column data and other table data for features respectively to obtain multiple other column feature vectors and other table feature vectors;

[0157] Using the feature vectors of the other columns and the feature vectors of the other tables, other data dictionaries to be improved are obtained;

[0158] The data dictionary is improved by using the other data dictionaries that need to be improved.

[0159] In one approach, such as Figure 5 The diagram shown illustrates the construction architecture flowchart for another dictionary proposed in this application. Specifically, it includes five steps: small sample data annotation, model building, model validation, data dictionary improvement, and replication of data dictionaries from the same industry.

[0160] For small sample data annotation,

[0161] First, the database maintainer can provide column field labels that cover all business implications and provide the corresponding column data (using these as sample column field labels and corresponding sample column data). As an example, each data table can provide one or more columns of data.

[0162] Additionally, database maintainers can provide a list of table topic tags covering all business themes, along with corresponding data (serving as sample table topic tags and corresponding sample table data). For example, each table topic tag could provide data for one or more tables.

[0163] Finally, in accordance with the dictionary specifications recorded by the data dictionary generation tool, the sample column field labels and associated sample column data, as well as the sample table topic labels and associated sample table data, are integrated to construct the initial data dictionary.

[0164] In one approach, the contents of the initial data dictionary are as follows: Figure 6 As shown. For example, only a small portion of the data is used for each label. Let N be the total number of column label categories and M be the total number of table topic label categories.

[0165] In terms of model building,

[0166] This application embodiment can construct an initial label classification model composed of a fully connected neural network. As an example, the initial label classification model includes four layers: an input layer (338 dimensions), a first hidden layer (128 dimensions), a second hidden layer (64 dimensions), and an output layer (N dimensions). The two hidden layers use the ReLU activation function, and the output layer uses the softmax activation function. Similarly, a similar fully connected neural network can be constructed for topic classification, the difference being that the output layer is M-dimensional.

[0167] Furthermore, in the training process of the initial label classification model in this embodiment, the cross-entropy loss, which is common to classification, can be used. The fastest optimizer, Adam, is selected, and the learning rate is set to a cosine annealing function. This is to prevent the learning rate from being too large when the loss is too small, which would lead to unstable results. The sample data is divided into an 80% training set and a 20% validation set. The multi-dimensional feature vectors of the columns and tables in the training set are input into the corresponding models. Training ends when the difference between the losses of two iterations is less than a specified threshold. Thus, the final target label classification model is obtained.

[0168] For model validation,

[0169] This application embodiment can prepare certain validation data, including but not limited to accuracy, precision, recall, and F1 score. This validation data is then input into the target label classification model trained in the second part. When the validation set meets the preset evaluation metrics, the model is saved. Otherwise, training of the model continues.

[0170] In particular, regarding the improvement of the data dictionary,

[0171] First, the embodiments of this application can obtain information about all tables and columns in the database, such as database type, instance name, table name, and column name. All table and column data are feature-encoded. 1,000 data points are randomly selected from each column, and 1,000 rows of data are randomly selected from each table. After feature encoding, these data form corresponding multi-dimensional feature vectors, which are used as input to the target label classification model.

[0172] Furthermore, in this embodiment, the aforementioned feature encoding can be input into the trained model, and each column and each table will output the corresponding classification result. This allows the obtained information and the corresponding classification results to be integrated using a data dictionary generation tool to form a data dictionary to be improved. This data dictionary is then concatenated with the initial data dictionary to form a complete data dictionary.

[0173] Specifically, regarding the replication of data dictionaries within the same industry,

[0174] In subsequent applications, for data from the same industry (i.e., other data dictionaries with the same business data type as the data dictionary), the other column data and other table data contained therein can be feature-encoded, and the pre-trained target label classification model can be used to directly output the column attributes and table topic classification results of the data, and then concatenate them into the data dictionary obtained in this application.

[0175] Understandably, the above steps avoid the time-consuming process of repeatedly training the model. This process assists business personnel in quickly labeling data table columns, enabling the efficient establishment of a unified data dictionary for the industry and freeing them from many limitations imposed by database software developers.

[0176] By applying the technical solution of this application, a preset character classification model can be used to automatically identify the column labels corresponding to feature-encoded column data, and automatically identify the table labels corresponding to feature-encoded table data. Then, a data dictionary generation tool is used to integrate the obtained data and the corresponding label classification results to form the final data dictionary. This achieves the goal of intelligently constructing a data dictionary, avoiding the time-consuming and labor-intensive drawbacks of manually building data dictionaries in existing technologies. Furthermore, because this application uses feature encoding technology that can completely extract the semantic information of the data, it can classify the data more meticulously to obtain a more accurate data dictionary.

[0177] Optionally, in another embodiment of this application, such as Figure 7 As shown, this application also provides a dictionary construction apparatus. It includes:

[0178] The acquisition module 201 is configured to acquire multiple column data sets and table data sets contained in the database, wherein the column data in each column data set belongs to the same column field label, and the table data in each table data set belongs to the same table topic label;

[0179] The encoding module 202 is configured to perform feature encoding on all column data contained in the plurality of column data sets to obtain a plurality of column feature vectors; and to perform feature encoding on all table data contained in the plurality of table data sets to obtain a plurality of table feature vectors.

[0180] The output module 203 is configured to input the plurality of column feature vectors and the plurality of table feature vectors into the pre-trained target label classification model to obtain the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector.

[0181] The construction module 204 is configured to construct a data dictionary based on the column field labels and the corresponding column data, and based on the table topic labels and the corresponding table data.

[0182] By applying the technical solution of this application, a preset character classification model can be used to automatically identify the column labels corresponding to feature-encoded column data, and automatically identify the table labels corresponding to feature-encoded table data. Then, a data dictionary generation tool is used to integrate the obtained data and the corresponding label classification results to form the final data dictionary. This achieves the goal of intelligently constructing a data dictionary, avoiding the time-consuming and labor-intensive drawbacks of manually building data dictionaries in existing technologies. Furthermore, because this application uses feature encoding technology that can completely extract the semantic information of the data, it can classify the data more meticulously to obtain a more accurate data dictionary.

[0183] In another embodiment of this application, the steps configured to be executed by the acquisition module 201 include:

[0184] Retrieve all column field labels and all table topic labels contained in the database;

[0185] The actual semantics corresponding to each column field label are parsed separately, and the actual semantics corresponding to each table topic label are also parsed separately.

[0186] Column field labels with the same actual semantic meaning from multiple column field labels are merged to obtain filtered column field labels; and table topic labels with the same actual semantic meaning from multiple table topic labels are merged to obtain filtered table topic labels.

[0187] Extract the multiple column data sets from the filtered column field labels; and extract the multiple table data sets from the filtered table topic labels.

[0188] In another embodiment of this application, the steps configured to be executed by the acquisition module 201 include:

[0189] The column data is feature-encoded according to a preset feature encoding rule to obtain the column feature vector; and the table data is feature-encoded according to the preset feature encoding rule to obtain the table feature vector;

[0190] The preset feature encoding rule includes at least one of the following encoding dimensions:

[0191] Dimensions include: the proportion of numeric characters to all characters; the proportion of time / name columns to all columns; the average number of characters per cell; the average number of numeric characters per cell; the standard deviation of the number of numeric characters per cell; whether it contains zero characters; whether it consists entirely of zero characters; the mean number of each digit from 0 to 9; whether it contains the letter 'a'; whether it consists entirely of 'a' characters; and the mean number of each letter 'az'.

[0192] In another embodiment of this application, the steps configured to be executed by the acquisition module 201 include:

[0193] Using a data dictionary generation tool, establish the first association between each column field label and its corresponding column data; and establish the second association between each table topic label and its corresponding table data.

[0194] According to the dictionary specifications recorded by the data dictionary generation tool, the first association relationship and the second association relationship are integrated to obtain the data dictionary to be improved;

[0195] The data dictionary to be improved is concatenated with the pre-built initial data dictionary to obtain the data dictionary.

[0196] In another embodiment of this application, the steps configured to be executed by the acquisition module 201 include:

[0197] Retrieve the labels of multiple sample column fields and the topic labels of sample tables contained in the database;

[0198] Each sample column field label is associated with at least one sample column data, and each sample table topic label is associated with at least one sample table data;

[0199] According to the dictionary specifications recorded by the data dictionary generation tool, the sample column field labels and associated sample column data are integrated, and the sample table topic labels and associated sample table data are integrated to construct the initial data dictionary.

[0200] In another embodiment of this application, the steps configured to be executed by the acquisition module 201 include:

[0201] The sample column field labels associated with the sample column data and the sample table topics associated with the sample table data are used as training data to train the initial label classification model until the target label classification model is obtained after training.

[0202] The target label classification model includes an input layer, a first hidden layer and a second hidden layer with ReLU activation function, and an output layer with softmax activation function.

[0203] In another embodiment of this application, the steps configured to be executed by the acquisition module 201 include:

[0204] Obtain at least one other data dictionary, wherein the other data dictionary and the data dictionary are dictionaries with the same business data type;

[0205] Extract multiple other column data and other table data contained in the other data dictionary, and encode the other column data and other table data for features respectively to obtain multiple other column feature vectors and other table feature vectors;

[0206] Using the feature vectors of the other columns and the feature vectors of the other tables, other data dictionaries to be improved are obtained;

[0207] The data dictionary is improved by using the other data dictionaries that need to be improved.

[0208] This application also provides an electronic device for performing the dictionary construction method described above. Please refer to... Figure 8This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 8 As shown, the electronic device 3 includes: a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected via the bus 302. The memory 301 stores a computer program that can run on the processor 300. When the processor 300 runs the computer program, it executes the dictionary construction method provided in any of the foregoing embodiments of this application.

[0209] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0210] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs. After receiving an execution instruction, the processor 300 executes the program. The data recognition method disclosed in any of the foregoing embodiments of this application can be applied to the processor 300, or implemented by the processor 300.

[0211] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.

[0212] The electronic device provided in this application embodiment and the data identification method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0213] This application also provides a computer-readable storage medium corresponding to the data recognition method provided in the foregoing embodiments. Please refer to... Figure 9 The computer-readable storage medium shown is an optical disc 40, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the dictionary construction method provided in any of the foregoing embodiments.

[0214] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0215] The computer-readable storage medium provided in the above embodiments of this application and the data identification method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0216] It should be noted that:

[0217] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0218] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0219] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0220] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing a dictionary, characterized in that, include: Retrieve multiple column data sets and table data sets contained in the database, where the column data in each column data set belongs to the same column field label, and the table data in each table data set belongs to the same table topic label; The feature encoding is performed on all column data contained in the multiple column data sets to obtain multiple column feature vectors; and the feature encoding is performed on all table data contained in the multiple table data sets to obtain multiple table feature vectors; the feature encoding is used to extract data semantic information; The multiple column feature vectors and the multiple table feature vectors are input into the pre-trained target label classification model to obtain the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector. A data dictionary is constructed based on the column field labels and their corresponding column data, and based on the table topic labels and their corresponding table data; The column data is feature-encoded according to a preset feature encoding rule to obtain the column feature vector; and the table data is feature-encoded according to the preset feature encoding rule to obtain the table feature vector; The preset feature encoding rule includes at least one of the following encoding dimensions: Dimensions include: the proportion of numeric characters to all characters; the proportion of time / name columns to all columns; the average number of characters per cell; the average number of numeric characters per cell; the standard deviation of the number of numeric characters per cell; whether it contains zero characters; whether it consists entirely of zero characters; the mean number of each digit from 0 to 9; whether it contains the letter 'a'; whether it consists entirely of 'a' characters; and the mean number of each letter 'az'.

2. The method as described in claim 1, characterized in that, The process of obtaining multiple column data sets and table data sets contained in the database includes: Retrieve all column field labels and all table topic labels contained in the database; The actual semantics corresponding to each column field label are parsed separately, and the actual semantics corresponding to each table topic label are also parsed separately. Column field labels with the same actual semantic meaning from multiple column field labels are merged to obtain filtered column field labels; and table topic labels with the same actual semantic meaning from multiple table topic labels are merged to obtain filtered table topic labels. Extract the multiple column data sets from the filtered column field labels; and extract the multiple table data sets from the filtered table topic labels.

3. The method as described in claim 1, characterized in that, The construction of a data dictionary based on the column field labels and corresponding column data, and based on the table topic labels and corresponding table data, includes: Using a data dictionary generation tool, establish the first association between each column field label and its corresponding column data; and establish the second association between each table topic label and its corresponding table data. According to the dictionary specifications recorded by the data dictionary generation tool, the first association relationship and the second association relationship are integrated to obtain the data dictionary to be improved; The data dictionary to be improved is concatenated with the pre-built initial data dictionary to obtain the data dictionary.

4. The method as described in claim 3, characterized in that, Before concatenating the data dictionary to be improved with the pre-built initial data dictionary, the method further includes: Retrieve the labels of multiple sample column fields and the topic labels of sample tables contained in the database; Each sample column field label is associated with at least one sample column data, and each sample table topic label is associated with at least one sample table data; According to the dictionary specifications recorded by the data dictionary generation tool, the sample column field labels and associated sample column data are integrated, and the sample table topic labels and associated sample table data are integrated to construct the initial data dictionary.

5. The method as described in claim 4, characterized in that, After obtaining the multiple sample column field labels and sample table topic labels contained in the database, the process also includes: The sample column field labels associated with sample column data and the sample table topic labels associated with sample table data are used as training data to train the initial label classification model until the target label classification model is obtained after training. The target label classification model includes an input layer, a first hidden layer and a second hidden layer with ReLU activation function, and an output layer with softmax activation function.

6. The method as described in claim 1 or 3, characterized in that, Following the construction of the data dictionary, the following is also included: Obtain at least one other data dictionary, wherein the other data dictionary and the data dictionary are dictionaries with the same business data type; Extract multiple other column data and other table data contained in the other data dictionary, and encode the other column data and other table data for features respectively to obtain multiple other column feature vectors and other table feature vectors; Using the feature vectors of the other columns and the feature vectors of the other tables, other data dictionaries to be improved are obtained; The data dictionary is improved by using the other data dictionaries that need to be improved.

7. A dictionary construction apparatus, characterized in that, include: The acquisition module is configured to acquire multiple column data sets and table data sets contained in the database, wherein the column data in each column data set belongs to the same column field label, and the table data in each table data set belongs to the same table topic label; The encoding module is configured to perform feature encoding on all column data contained in the multiple column data sets to obtain multiple column feature vectors; and to perform feature encoding on all table data contained in the multiple table data sets to obtain multiple table feature vectors; the feature encoding is used to extract data semantic information; The output module is configured to input the plurality of column feature vectors and the plurality of table feature vectors into the pre-trained target label classification model to obtain the column field label corresponding to each column feature vector and the table topic label corresponding to each table feature vector. The construction module is configured to construct a data dictionary based on the column field labels and corresponding column data, and based on the table topic labels and corresponding table data; perform feature encoding on the column data according to a preset feature encoding rule to obtain the column feature vector; and perform feature encoding on the table data according to the preset feature encoding rule to obtain the table feature vector; wherein the preset feature encoding rule includes at least one of the following encoding dimensions: the proportion of numeric characters to all characters, the proportion of time / name columns to all columns, the average number of characters per cell, the average number of numeric characters per cell, the standard deviation of the number of numeric characters per cell, whether it contains 0 characters, whether it is all 0 characters, the mean of the number of digits 0-9, whether it contains the 'a' character, whether it is all 'a' characters, and the mean of the number of letters 'az'.

8. An electronic device, characterized in that, include: Memory, used to store executable instructions; as well as, A processor, configured to execute the executable instructions with the memory to perform the operation of the dictionary construction method of any one of claims 1-6.

9. A computer-readable storage medium for storing computer-readable instructions, characterized in that, When the instruction is executed, it performs the operation of the dictionary construction method of any one of claims 1-6.

Images