Table relevance recommendation method based on multi-path recall and esim fine arrangement

Abstract

The application discloses a table relevance recommendation method based on multi-path recall and ESIM fine sorting, first, the input table is pretreated, the noise content in the table name and field is cleaned, and the input format of the model is arranged; then the jaccard similarity, the BM25 algorithm and the pre-training word vector model are used to carry out multi-path recall on the text of the input table, find several candidate tables most similar to the input table under each algorithm; the multiple recall results are combined, and a voting strategy is used to preliminarily coarsely sort; the coarsely sorted result is formatted into the model input and sent into the ESIM model for prediction, the model will sort the input finely, and output the table recommendation result according to the similarity from large to small. The application solves the technical problems of the deficiencies of the existing technology in judging semantic similarity, the errors caused by the word order, the mechanical process of text recommendation and the low efficiency.

Description

Technical Field

[0001] This invention mainly relates to the fields of text content similarity and recommendation algorithms in natural language processing. Background Technology

[0002] Current technology, when faced with an unknown input form, can only rely on natural human understanding and simple character matching to find similar form content. The steps are as follows:

[0003] 1. Observe the table name and field content, and extract keywords using business resources such as dictionaries.

[0004] 2. Use statistical methods or string comparison methods to find the table whose name is most similar to the input table.

[0005] 3. Use set similarity methods to compare the similarity between the field set of the input table and the field set of the standard table.

[0006] 4. Combine the results from steps 2 and 3 to output the complete standard table that is most similar to the input table.

[0007] This method, based on hard matching of text, typically leads to the following problems:

[0008] 1. Firstly, it does not consider the impact of word order on the entire text. For example, "Catalogue Version" and "Version Catalogue" have the key information "Version" and "Catalogue" respectively, but this word order information will be ignored during the matching process.

[0009] 2. Secondly, the biggest problem with hard matching is the omission of synonyms or semantically related texts in the matching (such as "school" and "education", "public security" and "police", etc.), which often results in similar texts not being reflected and output.

[0010] 3. Existing technologies are inefficient in practical applications, and their accuracy is often not guaranteed.

[0011] Therefore, due to the reasons explained above, the recommendation results for non-standardized or highly textual data may have significant errors. Summary of the Invention

[0012] To address the problems existing in the prior art and to overcome the shortcomings of traditional algorithms, this invention provides a table-based relevance recommendation method based on multi-path recall and ESIM ranking. This method solves the technical problems of insufficient ability to judge semantic similarity, errors caused by word order, and the mechanized process and low efficiency of text recommendation in the prior art.

[0013] To solve the above technical problems, the present invention adopts the following technical solution: a table relevance recommendation method based on multi-path recall and ESIM fine ranking, comprising the following steps:

[0014] Step S1: Preprocess the input table, clean and remove noisy content from the table name or fields, and organize it into the input format of the model.

[0015] Step S2: Use the Jaccard similarity coefficient, BM25 algorithm, and pre-trained word vector model to perform multi-way recall on the text of the input table, and obtain several candidate tables that are most similar to the input table under each algorithm.

[0016] Step S3: Merge the multiple recall results obtained in step S2 and use a voting strategy to perform preliminary coarse ranking.

[0017] Step S4: Format the coarse ranking results obtained in step S3 as model input and feed them into the ESIM model for prediction. The model performs fine ranking on the input and outputs the table recommendation results in descending order of similarity.

[0018] Furthermore, the specific process of performing multi-way recall of the text in the input table using the Jaccard similarity coefficient algorithm described in step S2 is as follows:

[0019] First, the keyword sets for the STG table and the ODS table are obtained by jieba word segmentation of the table name, Chinese field name, and data element identifier, respectively, and the keyword sets of the STG table and the ODS table are used as input.

[0020] Then, the Jaccard similarity coefficients J(A,B) of the STG table and all ODS are calculated using Equation (1) and sorted to obtain the top-k recall results with the highest similarity.

[0021]

[0022] In the formula, sets A and B represent the keyword sets of the STG table and the ODS table, respectively; when sets A and B are both... When J(A,B) = 1, we define J(A,B) = 1.

[0023] Furthermore, the pre-trained word vector algorithm described in step S2 assigns different weights to words based on their part of speech, obtains the vector representation of the text through weighted summation, and uses the vector cosine distance to measure the similarity between the tables.

[0024] Furthermore, the specific process of the pre-trained word vector algorithm is as follows:

[0025] (1) First, extract the table feature vector, generate feature vectors at different granularities according to the granularity, and concatenate them to form the representation vector of a single table; the representation vector includes table granularity and field granularity. Table granularity represents the Chinese name of the table and the table name overview, and field granularity represents the Chinese name of the field, the field description, the data source and the qualifying words.

[0026] (2) Internal weights: The table name and field parts are segmented into words and word vectors are obtained. Different part-of-speech weights are assigned to the word vectors according to their parts of speech. These weighted vectors are accumulated to obtain the corresponding table name vector and field vector.

[0027] (3) External weights: External weights are assigned different weights to the table name vector and field vectors, and the vector vec of the entire table is calculated using equation (2). table ,

[0028] vec table =w i *vec name +w j *vec field (2)

[0029] vec name A vector representation of the table name; w i Indicates the weight of the table name vector; vec field For the vector representation of the field; w j This indicates the weight of the field vector;

[0030] (4) Finally, the cosine similarity between the STG table representation vector and the representation matrix of the ODS candidate table set is calculated, and the top-k recall results are output from largest to smallest.

[0031] Furthermore, the specific process of the BM25 algorithm described in step S2 is as follows: Given the candidate sets of the STG table and the ODS table, the score between the ODS table and the document is obtained by calculating and accumulating the score of each word in the STG table. The higher the score, the higher the similarity. The top-k recall results are obtained by sorting the similarity from largest to smallest.

[0032] Furthermore, when calculating the score for each word in the STG, a full corpus model is used, which uses all word segmentation results of table names and fields as the corpus, and / or a table name model is used, which uses only the word segmentation results of table names as the corpus.

[0033] Furthermore, when the STG field is missing, the table name model is used; when the STG information is complete, both the table name model and the full corpus model are used to calculate the score, and the top-k recall results are obtained by sorting the scores from largest to smallest.

[0034] Furthermore, in step S3, all multiple recall results are initially screened according to statistical frequency, and the top-k retrieved results with the highest statistical frequency are selected as the candidate table set.

[0035] Furthermore, in step S4, the candidate table set that has undergone initial screening in step S3 is finely sorted based on the ESIM text ranking model. The specific method is as follows:

[0036] For the input layer, the original text pairs are adjusted into individual STGs and corresponding candidate sets, and the corresponding positive example indices are labeled; the loss function is the list-wise maximum positive example likelihood function, as shown in Equation (3):

[0037]

[0038] In the formula, logits is the final classification prediction value of the model, and targets is the true sorted sample label, which defaults to a list of length top_k, with 1 for positive examples and 0 for negative examples;

[0039] After model training, for the input STG table and the recalled ODS candidate table set, the output is the probability value corresponding to the ODS table at each position. The probability values ​​are sorted in descending order, which is the similarity ranking result.

[0040] Beneficial effects: Compared with the prior art, the present invention solves the technical problems of insufficient ability to judge semantic similarity, errors caused by word order, and the mechanized process and low efficiency of text recommendation in the prior art. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the recall method based on Jaccard similarity coefficient calculation as described in an embodiment of the present invention.

[0042] Figure 2 This is a flowchart illustrating the retrieval method based on word vector similarity calculation according to an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the table vector generation process described in an embodiment of the present invention;

[0044] Figure 4 This is a flowchart illustrating the retrieval method based on the bm25 algorithm as described in an embodiment of the present invention.

[0045] Figure 5 This is a schematic diagram of the ESIM network structure according to an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the final table recommendation process described in an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0048] Terminology Explanation:

[0049] STG table: This refers to the input non-standardized table, which is a text object that needs to be recommended.

[0050] ODS table: that is, the recommendation candidate table. In this invention, STG is recommended to certain candidate tables in ODS.

[0051] Jaccard coefficient: The Jaccard coefficient is a method for measuring the similarity of sets.

[0052] Embedding: Embedding representation (word / character vector) is a vector representation of text used for model computation.

[0053] BM25: Best Match algorithm, an algorithm in the field of text retrieval for evaluating the relevance between documents.

[0054] ESIM: Ehanced LSTM for Natural Language Inference, is a deep learning algorithm in the field of text matching.

[0055] LSTM (Long Short-Term Memory) is a type of temporal recurrent neural network that is widely used in most natural language processing tasks, such as text translation and document summarization.

[0056] Softmax: A commonly used function in deep learning, also known as a normalization function, which aims to represent multi-classification results in the form of probabilities.

[0057] Attention: The attention mechanism, also known as the attention mechanism, is widely used in deep learning. Its purpose is to make the model focus more on certain important features of the sample, invest more learning resources in these features, and suppress other useless information.

[0058] Data element: The core meaning of the Chinese names of table fields, provided by relevant resources.

[0059] Qualifier: The descriptive meaning of the Chinese name of the table field, which together with the data element constitutes the complete Chinese meaning of the field, provided by the relevant resources.

[0060] The purpose of this invention is to recommend the uploaded STG table and output the ODS table with the highest relevance.

[0061] Input: JSON data for STG, including the English name of the table, the Chinese name of the table, and field information.

[0062] Output: A list of ODS candidates most similar to the STG input.

[0063] To achieve the above objectives, this invention establishes a recommendation process combining recall and fine-grained ranking for text tasks involving text retrieval and ranking. Therefore, the overall design concept is divided into the following two stages:

[0064] I. Recall Phase

[0065] The recall phase utilizes three recall methods to retrieve results from the input STG table. The results are then coarsely ranked using a voting scheme, and the top k most similar results are output. The three recall methods are as follows:

[0066] a. Methods based on Jaccard coefficients

[0067] Given sets A and B, the Jaccard coefficient is defined as the ratio of the size of the intersection of A and B to the size of the size of the union of A and B, as shown in the following formula:

[0068]

[0069] In the formula, sets A and B represent the keyword sets of the STG table and the ODS table, respectively; when sets A and B are both... When J(A,B) = 1, we define J(A,B) = 1.

[0070] Specifically, in text recommendation tasks, the Jaccard similarity coefficient-based method takes the keyword sets of the STG and ODS tables as input. The keyword set consists of jieba segmentation of the table name, Chinese field names, and data element identifiers, serving as feature information representing a single table. By calculating the Jaccard similarity coefficients between the STG and all ODS tables and ranking them, the top k recall results with the highest similarity are obtained.

[0071] b. Weight-based word vector method

[0072] The weighted word vector method is based on a pre-trained word vector model. Different weights are assigned to words according to their part of speech. The vector representation of the text is obtained by weighted summation, and the similarity between the vectors is measured by the cosine distance between the vectors.

[0073] In extracting the vector representation of tables, since a single table contains a significant amount of Chinese information, the granularity is divided into table-level and field-level granularity. Feature vectors at different granularities are generated and concatenated to form the representation vector of a single table. The Chinese information at different granularities includes the following:

[0074] Table 1. Granularity Information Explanation

[0075]

[0076] like Figure 2 As shown, the weighted word vector method uses two parts, table name vector and field vector, to calculate similarity, and employs two weighting schemes, which are described in detail below:

[0077] 1. Static Weight:

[0078] The table name and field parts are segmented into words and word vectors are obtained. The word vectors are assigned different part-of-speech weights according to their parts of speech. These weighted vectors are accumulated to obtain the corresponding table name vector and field vector.

[0079] 2. External Weights (Dynamic Weight)

[0080] External weights are calculated by assigning different weights to the table name vector and field vectors, and then calculating the representation vector vec of the entire table. table The formula is as follows:

[0081] vec table =w i *vec name +w j *vec field (2)

[0082] vec name A vector representation of the table name; w i Indicates the weight of the table name vector; vec field For the vector representation of the field; w j This indicates the weight of the field vector;

[0083] As mentioned above, the vector generation process for a single table is as follows: Figure 3 Example:

[0084] The weighted word vector method calculates the cosine similarity between the STG table representation vector and the representation matrix of the ODS candidate table set, and outputs the top-k recall results in descending order of similarity.

[0085] c. Methods based on the bm25 algorithm

[0086] The BM25 algorithm is a common algorithm in the field of text retrieval based on a probabilistic retrieval model, used to evaluate the relevance between words and documents. In the context of this invention, the BM25 algorithm can be interpreted as follows: given a candidate set of STG and ODS tables, the score of each word 't' in the STG table is calculated and accumulated to obtain the score between the ODS table and the document; the higher the score, the higher the similarity.

[0087] The computational process applied to the table recommendation algorithm is as follows: Figure 4 The BM25 algorithm recall process is shown below:

[0088] Two BM25 models are used here: the full corpus model uses all word segmentation results of table names and fields as the corpus, while the table name model only uses word segmentation of table names. The choice between the two models depends on the input STG table.

[0089] 1) When the STG field is missing, only the table name model is used for calculation.

[0090] 2) When the STG information is complete, the scores will be calculated for both models, and the top k maximum scores will be merged and output.

[0091] Since the three methods focus on different aspects, the Jaccard coefficient-based method focuses on mining the degree of repetition between text sets to calculate Jaccard similarity; the word vector-based method pre-trains word vectors on a large-scale corpus and calculates Consine similarity based on representation vectors, focusing more on the semantic representation of table text; while the BM25 algorithm is a probabilistic retrieval method based on word frequency features, used to calculate the similarity between word t and STG and ODS and obtain a similarity score.

[0092] In summary, each method can be used as an independent recall module to output candidate results, or the recall results can be voted on, and the top-k results with the most votes can be used as input to the ranking model.

[0093] II. Text Sorting Stage

[0094] The text ranking model primarily uses ESIM as the fine-grained ranking model. ESIM (Ehanced LSTM for Natural Language Inference) is an LSTM model specifically designed for natural language inference. ESIM combines BiLSTM and attention mechanisms, taking paired texts as input to obtain a binary classification model for text matching. The output is mapped to the [0,1] interval through a softmax activation function, serving as a measure of text similarity. Its network structure is as follows... Figure 5 As shown:

[0095] After the recall phase calculation, 3*top_k recall results can be obtained, and the more similar the results are to the input table, the higher the repetition rate in the recall results. Therefore, the initial screening of the recall results adopts the statistical frequency method, selecting the top_k most frequent recall results as the candidate table set. The optimized ESIM is selected as the ranking model to perform fine ranking on the candidate set.

[0096] ESIM is a text matching model. The original model is essentially suitable for binary classification tasks. By adjusting the loss function and input layer, ESIM can be adapted to top-k classification tasks.

[0097] For the input layer, the original text pairs are refactored into individual STGs and corresponding candidate sets, with the corresponding positive example indices labeled. The loss function uses the list-wise maximum positive likelihood function, as shown in the following formula:

[0098]

[0099] In the formula, f is log_softmax; logits is the final classification prediction value of the model; targets is the true sorted sample label, which defaults to a list of length top_k, with 1 for positive examples and 0 for negative examples (e.g., [1,0,1,0,0] means that ODS at position 0 and 2 is a positive example, and the others are negative examples).

[0100] After model training, for the input STG table and the recalled ODS candidate table set, the model outputs the probability value corresponding to each position in the ODS table. These probabilities are then sorted in descending order to determine the similarity ranking. The final overall ranking process for table recommendations is as follows: Figure 6 As shown.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A table-based relevance recommendation method based on multi-path recall and ESIM ranking, characterized in that... Includes the following steps: Step S1: Preprocess the input table, clean and remove noisy content from the table name or fields, and organize it into the input format of the model. Step S2: Use the Jaccard similarity coefficient, BM25 algorithm, and pre-trained word vector algorithm to perform multi-way recall on the text of the input table, and obtain several candidate tables that are most similar to the input table under each algorithm. Step S3: Merge the various recall results obtained in step S2 and use a voting strategy to perform preliminary coarse ranking. Step S4: Format the coarse ranking results obtained in step S3 as model input and feed them into the ESIM model for prediction. The model performs fine ranking on the input and outputs the table recommendation results in descending order of similarity. The specific process of performing multi-way recall of the text in the input table using the Jaccard similarity coefficient algorithm in step S2 is as follows: First, the keyword sets for the STG table and the ODS table are obtained by jieba word segmentation of the table name, Chinese field name, and data element identifier, respectively, and the keyword sets of the STG table and the ODS table are used as input. Then, the Jaccard similarity coefficients of the STG table and all ODS tables are calculated using equation (1). The results are then sorted to obtain the top-k recall results with the highest similarity. （1）； In the formula, sets A and B represent the keyword sets of the STG table and the ODS table, respectively; when sets A and B are both... When, define ; The specific process of the pre-trained word vector algorithm is as follows: (1) First, extract the table feature vector, generate feature vectors at different granularities according to the granularity, and concatenate them to form the representation vector of a single table; the representation vector includes table granularity and field granularity. Table granularity represents the Chinese name of the table and the table name overview, and field granularity represents the Chinese name of the field, the field description, the data source and the qualifying words. (2) Internal weights: The table name and field parts are segmented into words and word vectors are obtained. Different part-of-speech weights are assigned to the word vectors according to their parts of speech. These weighted vectors are accumulated to obtain the corresponding table name vector and field vector. (3) External weights: External weights are assigned different weights to the table name vector and field vectors respectively, and the vector of the entire table is calculated by formula (2). , （2）； A vector representation of the table name; This indicates the weight of the table name vector; A vector representation of the field; This indicates the weight of the field vector; (4) Finally, the cosine similarity between the representation vectors of the STG table and the representation matrix of the ODS table is calculated, and the top-k recall results are output in descending order. In step S3, all multiple recall results are initially screened according to statistical frequency, and the top-k recall results with the highest statistical frequency are selected as the candidate table set. In step S4, the candidate table set that has undergone initial screening in step S3 is finely sorted based on the ESIM text ranking model. The specific method is as follows: For the input layer, the original text pairs are refactored into individual STGs and corresponding candidate sets, with corresponding positive example indices labeled; the loss function is the list-wise maximum positive likelihood function. As shown in equation (3): （3）； In the formula, This represents the model's final classification prediction value. Labels for the actual sorted samples. The default value is a list of length top_k, with 1 for positive examples and 0 for negative examples; Represents the logarithm Softmax; After model training, for the input STG table and the recalled ODS table, the probability value corresponding to each position of the ODS table is output, and the probability values ​​are sorted in descending order to obtain the similarity ranking result.

2. The table relevance recommendation method based on multi-path recall and ESIM fine ranking according to claim 1, characterized in that: The pre-trained word vector algorithm described in step S2 assigns different weights to words based on their part of speech, obtains the vector representation of the text through weighted summation, and uses the vector cosine distance to measure the similarity between the tables.

3. The table relevance recommendation method based on multi-path recall and ESIM fine ranking according to claim 1, characterized in that: The specific process of the BM25 algorithm described in step S2 is as follows: Given the STG table and the ODS table, the score of each word in the STG table is calculated and accumulated to obtain the score between the ODS table and the document. The higher the score, the higher the similarity. The top-k recall results are obtained by sorting the similarity from largest to smallest.

4. The table relevance recommendation method based on multi-path recall and ESIM fine ranking according to claim 3, characterized in that: When calculating the score for each word in the STG, a full corpus model is used, which uses all word segmentation results of table names and fields as the corpus, and / or a table name model is used, which uses only word segmentation results of table names as the corpus.

5. The table relevance recommendation method based on multi-path recall and ESIM fine ranking according to claim 4, characterized in that: When the STG field is missing, the table name model is used to calculate the score; when the STG information is complete, both the table name model and the full corpus model are used to calculate the score, and the top-k recall results are obtained by sorting the scores from largest to smallest.

Images