Methods, devices, storage media, and processors for querying similar text

Abstract

This application provides a method, apparatus, storage medium, and processor for querying similar text. The method includes: when the text type of the text to be queried is a first type of vocabulary, decomposing the text to be queried into multiple short words; determining a first target word vector for each short word; determining a second target word vector for the text to be queried based on the first target word vectors of all short words, thereby determining a first similarity between each word in the text database and the text to be queried; determining multiple first words in the text database that semantically match the text to be queried based on the first similarity, and a target similarity between each word in the text database and each first word; determining words in the text database whose target similarity is greater than a preset similarity threshold as target words matching the text to be queried. This greatly reduces the situation where text vectors cannot be recognized or have large errors, improving the success rate and accuracy of text recognition.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to a method, apparatus, storage medium, and processor for querying similar text. Background Technology

[0002] In the data warehouse of the construction machinery industry, inconsistencies in field naming conventions result in a large number of fields with different names, similar semantics, and identical actual meanings, leading to redundancy. During data warehouse operation and maintenance, similar fields can increase communication and integration costs, complicate data tracing, and increase the workload of data warehouse operation and maintenance, thereby affecting the user experience of front-end reports for business departments. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, storage medium, and processor for querying similar text.

[0004] To achieve the above objectives, the first aspect of this application provides a method for querying similar texts, comprising:

[0005] Determine the text type of the text to be queried;

[0006] When the text type is a type 1 word, the text to be queried is broken down into multiple short words;

[0007] Determine the first target word vector for each short word;

[0008] The second target word vector of the query text is determined based on the first target word vector of all short words.

[0009] The first similarity between each word in the text database and the text to be queried is determined based on the second target word vector.

[0010] Based on the first similarity, determine multiple first words in the text database that semantically match the text to be queried, and the target similarity between each word in the text database and each first word;

[0011] Words in the text database with a target similarity greater than a preset similarity threshold are identified as target words that match the text to be queried.

[0012] In embodiments of this application, determining the first target word vector for each short word includes: for each short word, if a first word vector exists for the short word, determining a second similarity between the short word and each word in the text database based on the first word vector and the second word vector of each word; for each short word, determining a first target word vector based on multiple second similarities; and for each short word, if a first word vector does not exist for the short word, determining a first target word vector based on the vector dimension of any word in the text database.

[0013] In embodiments of this application, determining a first target word vector for each short word based on multiple second similarities includes: for each short word, determining multiple second words in a text database that semantically match the short word based on multiple second similarities; and for each short word, determining a first target word vector based on multiple second word vectors of the multiple second words.

[0014] In the embodiments of this application, for each short word, determining multiple second words in the text database that semantically match the short word based on multiple second similarities includes: for each short word, sorting multiple words in the text database according to the size of the second similarity to determine the first word order of multiple words in the text database; for each short word, selecting a first preset number of words from the text database as multiple second words according to the word order, wherein the second words refer to the words that appear earlier in the first word order in the text database.

[0015] In embodiments of this application, determining a first target word vector for each short word based on multiple second word vectors of multiple second words includes: for each short word, determining the exponential value of the exponential smoothing coefficient corresponding to the second word vector of each second word according to the order of the first words; and for each short word, determining the first target word vector based on multiple exponential smoothing coefficients, multiple exponential values, and multiple second word vectors of multiple second words.

[0016] In embodiments of this application, determining the first target word vector for each short word based on multiple exponential smoothing coefficients, multiple exponential values, and multiple second word vectors of multiple second words includes:

[0017] The first target word vector is calculated according to the following formula (1):

[0018] v k = (1-β)×(θ0+β×θ1+β) 2 ×θ2+β 3 ×θ3+...+β n ×θ n (1)

[0019] Where vj refers to the first target word vector of the k-th short word, β refers to the exponential smoothing coefficient corresponding to the k-th short word, and θ0, θ1, θ2, θ3...θ n These refer to the second word vectors of the 1st, 2nd, 3rd, 4th...n+1th second words.

[0020] In the embodiments of this application, determining the text type of the text to be queried includes: if the text to be queried does not have a third word vector, determining the text to be queried as a first type of vocabulary; if the third word vector exists, determining the text to be queried as a second type of vocabulary, and determining the third word vector as the target word vector of the text to be queried.

[0021] In embodiments of this application, determining multiple first words in the text database that semantically match the text to be queried based on a first similarity, and the target similarity between each word in the text database and each first word includes: sorting the multiple words according to the magnitude of the first similarity to determine the second word order of the multiple words; and selecting a second preset number of words from the text database as multiple first words according to the second word order, wherein the first word refers to the word that appears first in the second word order in the text database.

[0022] In embodiments of this application, determining multiple first words in a text database that semantically match the text to be queried based on a first similarity, and determining the target similarity between each word in the text database and each first word, includes: determining a third similarity corresponding to the edit distance between each word in the text database and each first word; and for each word in the text database, determining a target similarity based on the first similarity and the third similarity between each word and each first word.

[0023] In embodiments of this application, determining the target similarity for each word in the text database based on the first similarity and the third similarity between each word and each first word includes: determining the product of each third similarity and the first similarity of each first word for each word in the text database; and determining the target similarity for each word in the text database based on the maximum value among multiple products and the first similarity.

[0024] In embodiments of this application, determining multiple first words in a text database that semantically match the text to be queried based on a first similarity, and the target similarity between each word in the text database and each first word includes: obtaining a preset similarity between each word and the text to be queried; and for each word in the text database, taking a weighted average of the first similarity, the maximum value among multiple products, and the preset similarity to determine the target similarity.

[0025] In embodiments of this application, determining the first similarity between each word in the text database and the text to be queried based on the second target word vector includes: determining the second word vector of each word in the text database; for each word, determining the first similarity based on the second target word vector and the second word vector of each word, wherein the first similarity is calculated according to the following formula (2):

[0026]

[0027] Where 'a' refers to the 'a'th word in the text database, 'b' refers to the text to be queried, and 'd' refers to the word in the text database. a，b This refers to the first similarity between the a-th word in the text database and the text to be queried, v a It refers to the second word vector of the a-th word, v b This refers to the second target word vector of the query text, a i It refers to the i-th element of the a-th word, b j It refers to the i-th element of the text to be queried.

[0028] A second aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned method for querying similar text.

[0029] A third aspect of this application provides a processor configured to execute the aforementioned method for querying similar text.

[0030] A fourth aspect of this application provides a similar text detection apparatus, including a processor configured to perform the aforementioned similar text query method.

[0031] The above-described method, device, storage medium, and processor for querying similar texts involves: determining the text type of the text to be queried; decomposing the text into multiple short words when the text type is a first type of vocabulary; determining the first target word vector for each short word; determining the second target word vector for the text to be queried based on the first target word vectors of all short words; determining the first similarity between each word in the text database and the text to be queried based on the second target word vectors; determining multiple first words in the text database that semantically match the text to be queried based on the first similarity, and determining the target similarity between each word in the text database and each first word; and identifying words in the text database whose target similarity is greater than a preset similarity threshold as target words that match the text to be queried. For complex, unfamiliar, spliced, or disordered texts with difficult-to-identify vectors, decomposing the text into multiple short words and performing semantic matching based on the word vectors of these short words, filtering semantically similar words, greatly reduces the situation where text vectors cannot be identified or have large errors, thus improving the success rate and accuracy of text recognition.

[0032] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0033] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0034] Figure 1 The illustration shows a flowchart of a method for querying similar texts according to an embodiment of this application;

[0035] Figure 2 The schematic diagram illustrates a flowchart of a method for querying similar texts according to yet another embodiment of this application;

[0036] Figure 3 This schematic diagram illustrates the structure of a similar text query device according to an embodiment of the present application;

[0037] Figure 4 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0039] Figure 1 The illustration shows a flowchart of a method for querying similar text according to an embodiment of this application. For example... Figure 1 As shown in one embodiment of this application, a method for querying similar text is provided, including the following steps:

[0040] S102, Determine the text type of the text to be queried.

[0041] S104, when the text type is a first type of vocabulary, decompose the text to be queried into multiple short words.

[0042] The query text refers to the text to be semantically matched. Semantic matching allows us to find words in the text database that are semantically identical or similar to the query text. Text types include Type I words, which are complex words whose word vectors cannot be calculated using existing computational models. In the engineering machinery industry, due to the large volume of documents and the presence of numerous technical terms, spliced ​​short texts, and difficult short texts within many words, many query texts cannot have their word vectors directly determined due to excessive length, disordered word order, or obscure language. Such words belong to Type I words. The processor can determine the text type of the query text.

[0043] If the text type is a first-class vocabulary, meaning the query text is a complex vocabulary, the processor can decompose it into multiple shorter words. Specifically, the processor can use Chinese word segmentation technology to remove stop words and meaningless words from the query text, thus reducing the interference of these words on semantic matching. For example, if the query text is A, A can be decomposed into the short words A1, A2, and A3.

[0044] S106, determine the first target word vector for each short word.

[0045] S108, determine the second target word vector of the query text based on the first target word vector of all short words.

[0046] The first target word vector refers to the target value of the word vectors of the short words. The processor can determine how to process each short word in the decomposition of the query text, thereby determining the first target word vector for each short word. Therefore, all the short words obtained from the decomposition of the query text correspond to multiple first target word vectors. The processor can determine the second target word vector of the query text based on the first target word vectors of all the short words. The second target word vector refers to the target value of the word vectors of the query text. If the word vectors of the query text cannot be directly calculated using existing models, the second target word vector of the query text can be indirectly calculated by using the first target word vectors of the short words decomposed from the query text. For example, if the query text is A, and A is decomposed into short words A1, A2, and A3, their corresponding first target word vectors are a1, a2, and a3. Then, the second target word vector a0 of A can be determined based on a1, a2, and a3.

[0047] S110, determine the first similarity between each word in the text database and the text to be queried based on the second target word vector.

[0048] The text database includes multiple words, each with a known word vector. In a specific implementation, the words in the text database can be loaded into a word2vec model to improve the speed of similar text queries. The first similarity refers to the cosine similarity between each word in the text database and the query text. Any two words can be semantically similar using cosine similarity. Specifically, the cosine of the angle between two vectors can be calculated to evaluate their similarity. Specifically, based on the second target word vector of the query text and the word vectors of each word in the text database, the processor can determine the similarity between these two word vectors and use this similarity as the first similarity between each word in the text database and the query text. For example, if the query text is A, and A is decomposed into short words A1, A2, A3, and the text database includes words B1, B2, B3, C1, C2, C3, then the second target word vector a0 of A can be used to further calculate multiple first similarities d between A1 and B1, A1 and B2, A1 and B3, etc. 11 d 12 d 13 ...

[0049] S112, determine multiple first words in the text database that semantically match the text to be queried based on the first similarity, and the target similarity between each word in the text database and each first word.

[0050] S114, words in the text database with a target similarity greater than a preset similarity threshold are identified as target words that match the text to be queried.

[0051] The first term refers to the word in the text database that semantically matches the query text. After calculating the first similarity between each word in the text database and the query text, the processor can filter out multiple first terms from the multiple words contained in the text database based on the first similarity. Furthermore, the processor can also calculate the target similarity between each word in the text database and each first term. The processor can filter the words in the text database using a preset similarity threshold corresponding to the query text, selecting words with a target similarity greater than the preset similarity threshold as target terms to match the query text. Here, target similarity refers to the target value of the similarity between the query text and each word in the text database. Target terms refer to words in the text database that are semantically the same as or similar to the query text. The preset similarity threshold is a similarity threshold set by the technician based on the query requirements. Different preset similarity thresholds can be set according to the query requirements; different thresholds result in different target terms and different numbers of target terms. Generally speaking, the higher the threshold, the fewer the number of target terms, and the closer the target terms are to the query text semantically.

[0052] For example, the first similarity d 11 d 12 d 13 ...In the given text, words B1 and B2 corresponding to d1 and d2 are the first words. Then, choosing any one word from B1, B2, B3, C1, C2, and C3, and any one word from B1 and B2, according to d... 11 and d 12 The target similarity D between the two words can be calculated. This yields multiple target similarities D1, D2, D3, and so on. Furthermore, values ​​of D1, D2, D3, etc., that are greater than a preset similarity threshold can be filtered out. If D1 and D2 are greater than the preset similarity threshold, then the words corresponding to D1 and D2 are the target words.

[0053] In one embodiment, determining the text type of the query text includes: if the query text does not have a third word vector, determining the query text as a first type of word; if the third word vector exists, determining the query text as a second type of word, and determining the third word vector as the target word vector of the query text.

[0054] The third word vector refers to the word vector of the query text, which is directly calculated using an existing computational model. More specifically, this computational model is a word2vec model trained on existing vocabulary samples. Query texts that are too long, have disordered word order, or are obscure cannot have their word vectors directly determined using existing computational models. If the third word vector of the query text cannot be directly obtained, then the query text is classified as a first-type vocabulary, i.e., complex vocabulary. If the third word vector can be directly obtained, then the query text is classified as a second-type vocabulary, and the third word vector is determined as the target word vector of the query text. Second-type vocabulary refers to vocabulary whose word vectors can be directly calculated using a computational model.

[0055] In one embodiment, a training step for the word2vec model is also included, comprising:

[0056] 1. Input multiple corpora (vocabularies). Since there is relatively little qualified corpus data in the engineering machinery industry that has been collected, Wikipedia corpus data is added as the main supplement.

[0057] 2. Scan multiple corpora, segment them using word segmentation technology, and then extract stems and restore word forms.

[0058] 3. Construct a dictionary and count the frequency of words in the dictionary.

[0059] 4. Construct a tree structure: build a Huffman tree according to the frequency of word occurrence, and generate the binary code of the nodes in the Huffman tree.

[0060] 5. Initialize the intermediate vectors of each non-leaf node and the word vectors in the leaf nodes.

[0061] 6. Train the intermediate vectors and word vectors to obtain the trained word2vec model.

[0062] In one embodiment, determining the first target word vector for each short word includes: for each short word, if a first word vector exists for the short word, determining a second similarity between the short word and each word in the text database based on the first word vector and a second word vector for each word; for each short word, determining a first target word vector based on multiple second similarities; and for each short word, if a first word vector does not exist for the short word, determining a first target word vector based on the vector dimension of any word in the text database.

[0063] In one embodiment, determining a first target word vector for each short word based on multiple second similarities includes: determining multiple second words in a text database that semantically match the short word based on multiple second similarities; and determining a first target word vector for each short word based on multiple second word vectors of the multiple second words.

[0064] The first word vector refers to the word vector of each short word, and the second word vector refers to the word vector of each word in the text database, both calculated directly using existing computational models. More specifically, this computational model is a word2vec model trained using existing vocabulary samples. The second similarity refers to the cosine similarity between each word in the text database and each short text. However, after decomposing the query text into multiple short words using Chinese word segmentation, there may still be cases where the word vectors of the decomposed short words cannot be directly obtained. If the first word vector of the short words can be directly obtained, the processor can calculate the second similarity based on the first and second word vectors, and then further calculate the first target word vector based on the second target similarity.

[0065] Specifically, for each short word, the processor can determine multiple second words in the text database that semantically match the short word based on multiple second similarities. Here, a second word refers to a word in the text database that semantically matches the short word. After calculating the second similarity between each word in the text database and the short word, the higher the value of the second similarity, the higher the semantic similarity between the corresponding word and the short word. Based on the second similarity, the processor can filter out multiple second words from the multiple words contained in the text database. Then, the processor can calculate the first target word vector based on the second word vector of the second word.

[0066] If the first word vector cannot be directly obtained, the processor can determine the first target word vector based on the vector dimension of any word in the text database. Specifically, if the first word vector of the short word cannot be output through the word2vec model, the similarity between the short word and each word in the text database cannot be calculated using word vectors. Therefore, the processor can directly generate a random vector value as the first target word vector for the short word, and the vector dimension of this random vector value must be the same as the word vectors of the words in the text database to facilitate further calculations. This solves the problem that the word vectors of the query text or short word cannot be directly obtained.

[0067] In one embodiment, for each short word, determining multiple second words in the text database that semantically match the short word based on multiple second similarities includes: for each short word, sorting multiple words in the text database according to the magnitude of the second similarity to determine the first word order of the multiple words in the text database; for each short word, selecting a first preset number of words from the text database as multiple second words according to the first word order, wherein the second words refer to the words in the text database that appear earlier in the first word order.

[0068] If the first word vector of a short word can be directly obtained, then the second similarity between the short word and the words in the text database can be calculated based on the first and second word vectors. For each short word, the words in the text database can be sorted according to the magnitude of the second similarity to obtain the first word order. The higher the value of the second similarity, the earlier the corresponding word appears in the order. The processor can filter out a first preset number of words according to the first word order as the second words semantically matching the short word, and further determine the first target word vector based on the second word vector of the second words. The first preset number is the number of second words set by technicians based on experience. For example, if the query text is A, A is decomposed into short words A1, A2, and A3, and the text database includes words B1, B2, B3, C1, C2, and C3. For the short word A1, its corresponding second similarity is d. 21 d 22 d 23 d 24 d 25 d 26 After sorting the words in the text database according to their size values, the first word order is C1, B1, C2, C3, B2, B3. If the first preset quantity is set to 3, then the second word that semantically matches this short word is C1, B1, C2. In practical applications, the first preset quantity is usually set to 10, which can maintain high accuracy while controlling the amount of computation.

[0069] In one embodiment, determining a first target word vector for each short word based on multiple second word vectors of multiple second words includes: for each short word, determining the exponential value of the exponential smoothing coefficient corresponding to the second word vector of each second word according to the order of the first words; and for each short word, determining the first target word vector based on multiple exponential smoothing coefficients, multiple exponential values, and multiple second word vectors of multiple second words.

[0070] If the first word vector of a short word can be directly obtained, then based on the first and second word vectors, a second similarity is calculated between the short word and words in the text database to filter out multiple second words. The processor can determine the exponential value of the exponential smoothing coefficient corresponding to the second word vector of the second word based on the order of the first word. A first target word vector is determined based on multiple exponential smoothing coefficients, multiple exponential values, and multiple second word vectors of multiple second words. Specifically, based on the order of the first word, the second word vectors of multiple second words form a vector sequence. An exponential moving average of this vector sequence with a window size of 5 can be calculated to determine the first target vector value of the short word. For example, the vector sequence might be θ0, θ1, θ2, θ3...θ n If the exponential smoothing coefficient β is 1, 2, 3, 4...n+1, then the exponential value of the corresponding exponential smoothing coefficient β is 1, 2, 3, 4...n+1.

[0071] In one specific embodiment, determining the first target word vector for each short word based on multiple exponential smoothing coefficients, multiple exponential values, and multiple second word vectors of multiple second words includes:

[0072] The first target word vector is calculated according to the following formula (1):

[0073] v k = (1-β)×(θ0+β×θ1+β) 2 ×θ2+β 3 ×θ3+...+β n ×θ n (1)

[0074] Among them, v i This refers to the first target word vector of the k-th short word, β refers to the exponential smoothing coefficient corresponding to the k-th short word, and θ0, θ1, θ2, θ3...θ n These refer to the second word vectors of the 1st, 2nd, 3rd, 4th...n+1th second words.

[0075] In one embodiment, determining the first similarity between each word in the text database and the text to be queried based on the second target word vector includes: determining the second word vector of each word in the text database; and for each word, determining the first similarity based on the second target word vector and the second word vector of each word, wherein the first similarity is calculated according to the following formula (2):

[0076]

[0077] Where 'a' refers to the 'a'th word in the text database, 'b' refers to the text to be queried, and 'd' refers to the word in the text database. a，b This refers to the first similarity between the a-th word in the text database and the text to be queried, v aIt refers to the second word vector of the a-th word, v b This refers to the second target word vector of the query text, a i It refers to the i-th element of the a-th word, b i It refers to the i-th element of the text to be queried.

[0078] In a specific embodiment, if the first word vector of the short word can be directly obtained, a second similarity between the short word and the words in the text database can be calculated based on the first and second word vectors. For each short word, the words in the text database can be sorted according to the magnitude of the second similarity to obtain a first word order. The higher the value of the second similarity, the earlier the corresponding word appears in the order. The processor can then select a first preset number of words according to the first word order as the second words that semantically match the short word.

[0079] Furthermore, based on the order of the first vocabulary, the second word vectors of multiple second vocabulary words form a vector sequence. The first target vector value of the short vocabulary word can be determined by calculating the exponential moving average of this vector sequence with a window size of 5. Specifically, the first target word vector of each short vocabulary word is calculated using the following formula (1) based on multiple exponential smoothing coefficients, multiple exponential values, and multiple second word vectors:

[0080] v k = (1-β)×(θ0+β×θ1+β) 2 ×θ2+β 3 ×θ3+...+β n ×θ n (1)

[0081] Among them, v i This refers to the first target word vector of the k-th short word, β refers to the exponential smoothing coefficient corresponding to the k-th short word, and θ0, θ1, θ2, θ3...θ n These refer to the second word vectors of the 1st, 2nd, 3rd, 4th...n+1th second words.

[0082] Furthermore, by combining the first target word vectors of all short words in the decomposed text to be queried, and taking their average, the second target word vector of the text to be queried is obtained. For each word in the text database, the processor can calculate the first similarity based on the second target word vector of the text to be queried and the second word vector of each word. The first similarity is calculated according to the following formula (2):

[0083]

[0084] Where 'a' refers to the 'a'th word in the text database, 'b' refers to the text to be queried, and 'd' refers to the word in the text database. a，bThis refers to the first similarity between the a-th word in the text database and the text to be queried, v a It refers to the second word vector of the a-th word, v b This refers to the second target word vector of the query text, a i It refers to the i-th element of the a-th word, b i This refers to the i-th element of the text to be queried. d a，b ∈[0,1],d a，b The closer the similarity is to 1, the more similar the meaning of the text b to be queried is to the meaning of the word a. It is understandable that the second similarity is relative to the first similarity. When the first word vector of the short word exists, the second similarity can be calculated according to the calculation method of the above calculation formula (2) based on the first word vector of the short word and the second word vector of each word in the text database.

[0085] In one embodiment, determining multiple first words in a text database that semantically match the text to be queried based on a first similarity, and the target similarity between each word in the text database and each first word includes: sorting the multiple words according to the magnitude of the first similarity to determine the second word order of the multiple words; and selecting a second preset number of words from the text database as multiple first words according to the second word order, wherein the first word refers to the word that appears first in the second word order in the text database.

[0086] After determining the second target word vector of the query text, the first similarity between each word in the text database and the query text can be calculated based on the second target word vector, thus obtaining multiple first similarities. The processor can then sort the words in the text database according to the magnitude of the first similarity, obtaining a second word order. The higher the first similarity value, the earlier the corresponding word appears in the order. The processor can then select a second preset number of words according to the second word order as the first words semantically matching the query text. The second preset number is the number of first words set by technicians based on experience. For example, the query text is A, and the text database includes words B1, B2, B3, C1, C2, and C3. Their corresponding first similarity is d. 11 d 12 d 13 d 14 d 15 d 16 After sorting the words in the text database according to their size values, the second word order is C1, B1, C2, C3, B2, B3. If the second preset quantity is set to 3, the first word that semantically matches the text to be queried is C1, B1, C2. In practical applications, the second preset quantity is usually set to 10, which can maintain high accuracy while controlling the amount of computation.

[0087] In one embodiment, determining multiple first words that are semantically matched with the text to be queried in the text database according to the first similarity, and the target similarity between each word in the text database and each first word includes: determining a third similarity corresponding to the edit distance between each word in the text database and each first word; for each word in the text database, determining the target similarity according to the first similarity and the third similarity between each word and each first word.

[0088] In one embodiment, for each word in the text database, determining the target similarity according to the first similarity and the third similarity between each word and each first word includes: for each word in the text database, determining the product of each third similarity and the first similarity of each first word; for each word in the text database, determining the target similarity according to the maximum value among multiple products and the first similarity.

[0089] When calculating text similarity, semantics is an aspect that people usually consider, but the character composition is also very important. The edit distance (Levenshtein distance) refers to the minimum number of edit operations required to convert one string into another between two strings. Dividing the edit distance by the maximum length between the two can obtain the similarity corresponding to the edit distance. When calculating the edit distance between two words, when the meanings are very similar but the character compositions are quite different, the result of the edit distance coefficient will be too low and distorted, resulting in a large difference in the calculation results. For example, for "吃饭" and "进食", the edit distance coefficient is 0. Directly calculating the edit distance coefficient will cause a large error. If the "intrinsic" and "extrinsic" can be combined, the result of the text similarity will be more persuasive and more stable. That is, for each word in the text database and the text to be queried, after calculating the first similarity corresponding to the cosine similarity between the two, it is also necessary to calculate the third similarity corresponding to the edit distance. Based on the first similarity and the third similarity, the similarity improved on the basis of semantic matching can be determined. The third similarity refers to the similarity corresponding to the edit distance between each word in the text database and each first word.

[0090] Specifically, the processor can sort the words in the text database according to their first similarity score, thus determining a second word order. The higher the first similarity score, the earlier the corresponding word appears in the order. The processor can then select a second preset number of words according to this second word order as the first words for semantic matching with the query text. For each first word and each word in the text database, a third similarity score is calculated, and the first and third similarities are multiplied to obtain a fourth similarity score. This fourth similarity score is an improved similarity score based on the edit distance, building upon semantic matching. Therefore, multiple improved fourth similarities can be obtained for multiple words in the text database and multiple first words. The processor can then select the maximum value from these products as the fifth similarity score corresponding to the query text. Compared to directly calculating the edit distance similarity between the query text and the words in the text database, the fifth similarity score incorporates cosine similarity, calculating edit distance similarity based on semantic matching. Furthermore, the maximum value from the product and the first similarity score are incorporated into a comprehensive calculation to further determine the target similarity. This can improve the success rate and accuracy of text recognition and matching.

[0091] For example, if the text to be queried is A, and the first word in the text database that semantically matches the text to be queried is r. a1 r a2 r a3 ...r a10 Its corresponding first similarity d a1 d a2 d a3 ...d a10 The third similarity is h a1 h a2 h a3 ...h a10 The higher the third similarity score, the more similar the character compositions. Multiplying the first similarity score by the corresponding third similarity score yields h. a1 r a1 h a2 r a2 h a3 r a2 ...h a10 r a2 The similarity scores are then sorted by value, and the maximum value of the product is taken as the fifth similarity score corresponding to the query text. A weighted average of the maximum value in the product and the first similarity score is then calculated to further determine the target similarity. The weights corresponding to the similarity scores can be determined experimentally.

[0092] In one embodiment, determining multiple first words in a text database that semantically match the text to be queried based on a first similarity, and the target similarity between each word in the text database and each first word includes: obtaining a preset similarity between each word and the text to be queried; and for each word in the text database, taking a weighted average of the first similarity, the maximum value among multiple products, and the preset similarity to determine the target similarity.

[0093] Business personnel can define similarity relationships between fields based on their business knowledge. Preset similarity scores are directly set by business personnel and incorporated into the similarity calculation of the query text. The processor can then calculate the target similarity between each word and the query text by weighting the first similarity score, the maximum value of multiple products, and the preset similarity score. Incorporating custom similarity relationships added based on business understanding into the comprehensive similarity calculation makes the recognition more flexible and better reflects actual business needs.

[0094] Figure 2 A schematic flowchart illustrating a method for querying similar text according to yet another embodiment of this application is shown. Figure 2 As shown in one embodiment of this application, a method for querying similar texts is provided. Specifically, semantic training materials can be collected first. Since the amount of qualified corpus data from the engineering machinery industry that has actually been collected is relatively small, Wikipedia corpus data is added as a major supplement. The processor can input multiple corpora into the word2vec model to train the word2vec model using multiple corpora. These semantic training materials include vocabulary from a text database.

[0095] The query text is input into the trained word2vec model. If the model outputs the third word vector of the query text, this third word vector is determined as the target word vector of the query text. If the model cannot output the third word vector, the query text is decomposed into multiple short words using Chinese word segmentation technology. Further, the processor can input each short word into the word2vec model. If the model outputs the first word vector of the short word, it calculates the second similarity between the two words using the first word vector and the second word vectors of words in the text database. For each short word, the processor can select the top ten words with the highest second similarity values ​​as the second words in the text database that are semantically similar to that short word. Based on the second word vectors of these ten second words, the processor can calculate the first target word vector of that short word. If the word2vec model cannot output the second word vector of the short word, the processor can generate a random vector value based on the word vectors of words in the text database as the first target word vector of the short word. The vector dimension of this random vector value is the same as the word vectors of words in the database to facilitate subsequent calculations. After obtaining the first target word vector for each short word, the processor can use the average of the first target word vectors of all short words as the second target word vector for the query text. Then, the processor can calculate the first similarity between the second target word vector and the second word vector of each word in the text database, which is the improved word2vec similarity coefficient.

[0096] Furthermore, after calculating the first similarity between the query text and each word in the text database, the top ten words with the highest first similarity values ​​can be considered as the first words in the text database that are semantically similar to the query text. The processor can then calculate the third similarity between each first word and each word in the text database, i.e., the similarity corresponding to edit distance. Based on the first and third similarities, the improved edit distance similarity, derived from semantic matching, can be determined. Specifically, the first and third similarities can be multiplied, and the processor can select the maximum value in the product as the fifth similarity corresponding to the query text. Compared to directly calculating the edit distance similarity between the query text and words in the text database, the fifth similarity incorporates cosine similarity, calculating the edit distance similarity based on semantic matching. Furthermore, the maximum value in the product and the first similarity are incorporated into a comprehensive calculation to further determine the target similarity. This improves the success rate and accuracy of text recognition and matching.

[0097] Building upon the above approach, business personnel can define similarity relationships between fields based on their business knowledge, setting preset similarity scores for each field and incorporating them into the similarity calculation of the query text. The processor can then calculate the target similarity between each word and the query text by weighting the first similarity score, the maximum value of multiple products, and the preset similarity score. Incorporating custom similarity relationships added based on business understanding into the comprehensive similarity calculation, and combining the similarity coefficients of short texts, makes the recognition more flexible and better aligned with actual business needs.

[0098] Furthermore, the processor can filter words in the text database (i.e., the data warehouse field library) by using a preset similarity threshold corresponding to the text to be queried, and select words with a target similarity greater than the preset similarity threshold as target words to match the text to be queried.

[0099] Based on the target keywords obtained from the above query, business personnel can use the results as a reference for designing and standardizing the naming of newly added fields. Furthermore, by combining short text similarity matching methods and data lineage tools, post-processing field tracing can be performed, optimizing and transforming the data processing workflow.

[0100] The above-described method, apparatus, storage medium, and processor for querying similar texts involves: determining the text type of the text to be queried; decomposing the text into multiple short words when the text type is a first type of vocabulary; determining the first target word vector for each short word; determining the second target word vector for the text to be queried based on the first target word vectors of all short words; determining the first similarity between each word in the text database and the text to be queried based on the second target word vectors; determining multiple first words in the text database that semantically match the text to be queried based on the first similarity, and the target similarity between each word in the text database and each first word; and determining words in the text database whose target similarity is greater than a preset similarity threshold as target words that match the text to be queried. Based on a limited training corpus, for complex, unfamiliar, spliced, or disordered texts where vectors are difficult to identify, the text is decomposed into multiple short words. Word vectors for these short words are obtained using operations such as word segmentation and exponential moving average of nearest neighbors. Cosine similarity is calculated based on the word vectors to perform semantic matching and filter semantically similar words. Based on the calculated cosine similarity, this solution improves the calculation of similarity corresponding to edit distance using cosine similarity, thereby obtaining target similarity. This enriches the connotation of text similarity, makes similarity relationship identification more accurate, and enhances stability. It significantly reduces instances where text vectors cannot be identified or have large errors, improving the success rate and accuracy of text recognition. Furthermore, custom similarity relationships added based on business understanding are incorporated into the comprehensive similarity calculation, making recognition more flexible and better aligned with business realities. Finally, this solution combines short text similarity matching methods with tools such as data lineage, proposing multiple application scenarios to effectively reduce the redundancy of data indicators, demonstrating high application value in data governance. Based on the above methods, this solution can provide a reference for the design of standardized new field naming, enable post-event field traceability, and optimize and transform data processing workflows.

[0101] Figure 1 This is a flowchart illustrating a method for querying similar text in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0102] In one embodiment, such as Figure 3 As shown, a similar text query device is provided, including a text type determination module, a text decomposition module, a word vector determination module, a first similarity determination module, a first vocabulary and target similarity determination module, and a target vocabulary filtering module, wherein:

[0103] The text type determination module 302 is used to determine the text type of the text to be queried.

[0104] The text decomposition module 304 is used to decompose the text to be queried into multiple short words when the text type is the first type of vocabulary.

[0105] The word vector determination module 306 is used to determine the first target word vector for each short word and to determine the second target word vector for the text to be queried based on the first target word vectors of all short words.

[0106] The first similarity determination module 308 is used to determine the first similarity between each word in the text database and the text to be queried based on the second target word vector.

[0107] The first word and target similarity determination module 310 is used to determine multiple first words in the text database that semantically match the text to be queried, and the target similarity between each word in the text database and each first word, based on the first similarity.

[0108] The target word filtering module 312 is used to determine the words in the text database whose target similarity is greater than a preset similarity threshold as target words that match the text to be queried.

[0109] In one embodiment, the word vector determination module 306 is further configured to, for each short word, determine a second similarity between the short word and each word in the text database based on the first word vector and the second word vector of each word, if a first word vector exists for the short word; determine a first target word vector based on multiple second similarities for each short word; and determine a first target word vector based on the vector dimension of any word in the text database if a first word vector does not exist for each short word.

[0110] In one embodiment, the word vector determination module 306 is further configured to, for each short word, determine a plurality of second words in the text database that semantically match the short word based on a plurality of second similarities; and for each short word, determine a first target word vector based on a plurality of second word vectors of the plurality of second words.

[0111] In one embodiment, the word vector determination module 306 is further configured to determine multiple second words in the text database that semantically match the short words based on multiple second similarities, including: for each short word, sorting the multiple words in the text database according to the size of the second similarity to determine the first word order of the multiple words in the text database; for each short word, selecting a first preset number of words from the text database as multiple second words according to the word order, wherein the second words refer to the words in the text database that appear earlier in the first word order.

[0112] In one embodiment, the word vector determination module 306 is further configured to, for each short word, determine the exponential value of the exponential smoothing coefficient corresponding to the second word vector of each second word according to the order of the first words; and for each short word, determine the first target word vector according to multiple exponential smoothing coefficients, multiple exponential values ​​and multiple second word vectors of multiple second words.

[0113] In one embodiment, the word vector determination module 306 is further configured to determine the first target word vector based on multiple exponential smoothing coefficients, multiple exponential values, and multiple second word vectors of multiple second words, including:

[0114] The first target word vector is calculated according to the following formula (1):

[0115] v k = (1-β)×(θ0+β×θ1+β) 2 ×θ2+β 3 ×θ3+...+β n ×θ n (1)

[0116] Among them, v i This refers to the first target word vector of the k-th short word, β refers to the exponential smoothing coefficient corresponding to the k-th short word, and θ0, θ1, θ2, θ3...θ n These refer to the second word vectors of the 1st, 2nd, 3rd, 4th...n+1th second words.

[0117] In one embodiment, the text type determination module 302 is further configured to determine the text to be queried as a first type of word when the third word vector does not exist; and to determine the text to be queried as a second type of word when the third word vector exists, and to determine the third word vector as the target word vector of the text to be queried.

[0118] In one embodiment, the first word and target similarity determination module 310 is further configured to sort multiple words according to the size of the first similarity to determine the second word order of the multiple words; and to select a second preset number of words from the text database according to the second word order as multiple first words, wherein the first words refer to the words in the second word order in the text database.

[0119] In one embodiment, the root first word and target similarity determination module 310 is further configured to determine the third similarity corresponding to the edit distance between each word in the text database and each first word; and for each word in the text database, determine the target similarity based on the first similarity and the third similarity between each word and each first word.

[0120] In one embodiment, the first word and target similarity determination module 310 is further configured to determine, for each word in the text database, the product of each third similarity and the first similarity of each first word; and for each word in the text database, determine the target similarity based on the maximum value among multiple products and the first similarity.

[0121] In one embodiment, the first word and target similarity determination module 310 is further configured to obtain a preset similarity between each word and the text to be queried; and for each word in the text database, to perform a weighted average of the first similarity, the maximum value among multiple products, and the preset similarity to determine the target similarity.

[0122] In one embodiment, the first similarity determination module 308 is further configured to determine the second word vector of each word in the text database; for each word, a first similarity is determined based on the second target word vector and the second word vector of each word, and the first similarity is calculated according to the following formula (2):

[0123]

[0124] Where 'a' refers to the 'a'th word in the text database, 'b' refers to the text to be queried, and 'd' refers to the word in the text database. a，b This refers to the first similarity between the a-th word in the text database and the text to be queried, v a It refers to the second word vector of the a-th word, v b This refers to the second target word vector of the query text, a i It refers to the i-th element of the a-th word, b i It refers to the i-th element of the text to be queried.

[0125] The similar text query device includes a processor and a memory. The text type determination module, text decomposition module, word vector determination module, first similarity determination module, first vocabulary and target similarity determination module, and target vocabulary filtering module are all stored as program units in the memory. The processor executes the above program modules stored in the memory to implement the corresponding functions.

[0126] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and methods for searching similar text can be implemented by adjusting kernel parameters.

[0127] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0128] This application provides a storage medium storing a program that, when executed by a processor, implements the aforementioned method for querying similar text.

[0129] This application provides a processor for running a program, wherein the program executes the above-described method for querying similar text.

[0130] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor A01, a network interface A02, memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The database stores data on similar text query methods. The network interface A02 communicates with external terminals via a network connection. When the processor A01 executes the computer program B02, it implements a similar text query method.

[0131] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] This application provides an apparatus, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the above-described method for querying similar text.

[0133] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform the program that initializes the query method steps for having the above-described similar text.

[0134] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0139] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0140] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0142] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for querying similar texts, characterized in that, The method includes: Determine the text type of the text to be queried; If the text type is a first type of vocabulary, the text to be queried is decomposed into multiple short words; Determine the first target word vector for each short word; The second target word vector of the query text is determined based on the first target word vector of all short words. The first similarity between each word in the text database and the text to be queried is determined based on the second target word vector; Based on the first similarity, determine a plurality of first words in the text database that semantically match the text to be queried, and the target similarity between each word in the text database and each first word; Words in the text database with a target similarity greater than a preset similarity threshold are identified as target words that match the text to be queried. The determination of the first target word vector for each short word includes: For each short word, if a first word vector exists for the short word, a second similarity between the short word and each word in the text database is determined based on the first word vector and a second word vector for each word. For each short word, the first target word vector is determined based on multiple second similarities; For each short word, if the first word vector does not exist for the short word, the first target word vector is determined based on the vector dimension of any word in the text database; Determining the first target word vector for each short word based on multiple second similarities includes: For each short word, multiple second words that semantically match the short word in the text database are determined based on multiple second similarities; For each short word, the first target word vector is determined based on multiple second word vectors of the multiple second words; The step of determining multiple first words in the text database that semantically match the text to be queried based on the first similarity, and the target similarity between each word in the text database and each first word includes: Determine the third similarity corresponding to the edit distance between each word in the text database and each first word; For each word in the text database, the target similarity is determined based on the first similarity and the third similarity between each word and each first word; The step of determining the target similarity for each word in the text database, based on a first similarity and a third similarity between each word and each first word, includes: For each word in the text database, determine the product of each third similarity and the first similarity of each first word; For each word in the text database, the target similarity is determined based on the maximum value among multiple products and the first similarity.

2. The method for querying similar texts according to claim 1, characterized in that, For each short word, determining multiple second words in the text database that semantically match the short word based on multiple second similarities includes: For each short word, multiple words in the text database are sorted according to the second similarity to determine the first word order of the multiple words in the text database; For each short word, a first preset number of words are selected from the text database according to the order of the first words to form the plurality of second words, wherein the second words refer to the words that appear first in the order of the first words in the text database.

3. The method for querying similar texts according to claim 2, characterized in that, The step of determining the first target word vector for each short word based on multiple second word vectors of the multiple second words includes: For each short word, the exponential value of the exponential smoothing coefficient corresponding to the second word vector of each second word is determined according to the order of the first words; For each short word, the first target word vector is determined based on multiple exponential smoothing coefficients, multiple exponential values, and multiple second word vectors of multiple second words.

4. The method for querying similar texts according to claim 3, characterized in that, The step of determining the first target word vector for each short word based on multiple exponential smoothing coefficients, multiple exponential values, and multiple second word vectors of multiple second words includes: The first target word vector is calculated according to the following formula (1): （1） in, This refers to the first target word vector of the k-th short word. It refers to the exponential smoothing coefficient corresponding to the k-th short word. , , They refer to the 1st, 2nd, 3rd, and 4th respectively. The second word vector of the (n+1)th second word.

5. The method for querying similar texts according to claim 1, characterized in that, The text type for determining the text to be queried includes: If the text to be queried does not contain a third word vector, the text to be queried is determined to be a first type of vocabulary; If the third word vector exists, the text to be queried is determined to be a second type of vocabulary, and the third word vector is determined as the target word vector of the text to be queried.

6. The method for querying similar texts according to claim 1, characterized in that, The step of determining multiple first words in the text database that semantically match the text to be queried based on the first similarity, and the target similarity between each word in the text database and each first word includes: The multiple words are sorted according to the first similarity to determine the second word order of the multiple words; According to the second word order, a second preset number of words are selected from the text database to form the plurality of first words, wherein the first words refer to the words that appear first in the second word order in the text database.

7. The method for querying similar texts according to claim 1, characterized in that, The step of determining multiple first words in the text database that semantically match the text to be queried based on the first similarity, and the target similarity between each word in the text database and each first word includes: Obtain the preset similarity between each word and the text to be queried; For each word in the text database, the first similarity, the maximum value among the multiple products, and the preset similarity are weighted and averaged to determine the target similarity.

8. The method for querying similar texts according to claim 1, characterized in that, The step of determining the first similarity between each word in the text database and the text to be queried based on the second target word vector includes: Determine the second word vector for each word in the text database; For each word, the first similarity is determined based on the second target word vector and the second word vector of each word, and the first similarity is calculated according to the following formula (2): （2） in, It refers to the a-th word in the text database. This refers to the text to be queried. This refers to the first similarity between the a-th word in the text database and the text to be queried. It refers to the second word vector of the a-th word. This refers to the second target word vector of the text to be queried. It refers to the i-th element of the a-th word. It refers to the i-th element of the text to be queried.

9. A processor, characterized in that, It is configured to perform a similar text query method according to any one of claims 1 to 8.

10. A similar text query device, characterized in that, Includes the processor as described in claim 9.

11. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform a method for querying similar texts according to any one of claims 1 to 8.

Images