Data search method for data dictionary and data center system
By performing word embedding operations on the input vocabulary to generate word vectors, and calculating the degree of correlation with the vocabulary in the data dictionary, the problem of time-consuming and poor adaptability of the existing data dictionary query methods is solved, and automated query and efficient management are realized.
Patent Information
- Application Number
- CN202311087223.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-25
- Publication Date
- 2025-03-04
AI Technical Summary
The establishment and query methods of existing data dictionaries require manual setting, which makes them time-consuming and labor-intensive, difficult to adapt to new or expanding data, lack adaptability, and as the number of rules increases, management and maintenance become complicated.
By performing word embedding operations on the input vocabulary to generate word vectors, calculate the correlation degree between the input vocabulary and multiple vocabulary in the data dictionary, use cosine similarity to judge recommended synonyms, and reduce the dependence on manually setting synonyms and conjunctions.
Automatic data dictionary query is realized, which improves query efficiency and accuracy, is more adaptable, and reduces the complexity of manual management and maintenance, especially as data continues to expand.
Smart Images
Figure CN119557411A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a data search method and a data center system for a data dictionary, and in particular to a data search method and a data center system for a data dictionary capable of providing recommended synonyms. Background Art
[0002] At present, the vocabulary in the data dictionary is usually built based on specific rules, and synonyms and associated words are all strongly linked. Users need to manually enter the set vocabulary and the synonyms and associated words of the vocabulary in the data dictionary interface to make the set vocabulary and its synonyms and associated words have a strong connection relationship. Due to the strong connection relationship, when the user searches, it is easy to have vocabulary mismatch results, that is, when the vocabulary entered by the user during the query is different from the vocabulary set in the data dictionary, no query results will be output. In short, the traditional data dictionary construction query method requires manual setting and is time-consuming and laborious. In addition, the data dictionary based on specific rules needs to query vocabulary through predefined rules, so it is difficult to adapt to new or expanding data and lacks adaptability. Moreover, as the number of rules increases, management and maintenance will become more complicated and the rule base must be expanded to adapt to search queries for large amounts of data. Therefore, the existing technology really needs to be improved. Summary of the invention
[0003] In order to solve the above problems, the present invention provides a data search method for a data dictionary and a data center system that can provide recommended synonyms to solve the above problems.
[0004] The present invention provides a data search method for a data dictionary, the method comprising: obtaining an input word and performing a word embedding operation on the input word to generate a word vector of the input word; calculating the degree of association between the input word and a plurality of words in the data dictionary according to the word vector of the input word, wherein the steps include: calculating a cosine similarity between the word vector of the input word and each of the plurality of word vectors of the plurality of words; and judging at least one recommended synonym from the plurality of words in the data dictionary according to the degree of association between the input word and the plurality of words in the data dictionary, wherein the steps include: when it is judged that the cosine similarity between the word vector of the input word and a word vector of a first word among the plurality of words in the data dictionary is greater than a threshold value, judging the first word as the recommended synonym.
[0005] The present invention further provides a data center system, which includes: a data dictionary, including a plurality of words; a processing circuit, coupled to the data dictionary, for obtaining an input word and performing a word embedding operation on the input word to generate a word vector of the input word; wherein the processing circuit calculates the degree of association between the input word and the plurality of words in the data dictionary based on the word vector of the input word, and determines at least one recommended synonym from the plurality of words in the data dictionary based on the degree of association between the input word and the plurality of words in the data dictionary, wherein the processing circuit calculates a cosine similarity between the word vector of the input word and each of the plurality of word vectors of the plurality of words, and when it is determined that the cosine similarity between the word vector of the input word and the word vector of a first word in the plurality of words in the data dictionary is greater than a threshold value, the processing circuit determines the first word as the recommended synonym. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1 Schematic diagram of a data center system in an embodiment of the present invention.
[0007] Figure 2 It is a schematic diagram of a process of an embodiment of the present invention.
[0008] Figure 3 It is a schematic diagram of the word vector of the input vocabulary or vocabulary in the data dictionary in three-dimensional space according to an embodiment of the present invention.
[0009] Figure 4 This is a schematic diagram of an embodiment of the recommended synonyms of the embodiment of the present invention.
[0010] Component number description
[0011] 1 Data Center System
[0012] 102 Data dictionary
[0013] 104 Processing circuit
[0014] 20 Process
[0015] Steps S200, S202, S204, S206, S208 DETAILED DESCRIPTION
[0016] Certain words are used in the specification and subsequent patent applications to refer to specific components. Those with ordinary knowledge in the field should understand that manufacturers may use different terms to refer to the same component. This specification and subsequent patent applications do not use differences in names as a way to distinguish components, but use differences in the functions of components as the basis for distinction. The "include" or "comprising" mentioned throughout the specification and subsequent patent applications is an open-ended term and should be interpreted as "including but not limited to". In addition, the term "coupled" herein includes any direct and indirect electrical connection means. Therefore, if the text describes a first device coupled to a second device, it means that the first device can be directly electrically connected to the second device, or indirectly electrically connected to the second device through other devices or connection means.
[0017] Please refer to Figure 1 , Figure 1 1 is a schematic diagram of a data center system 1 according to an embodiment of the present invention. The data center system 1 includes a data dictionary 102 and a processing circuit 104. The data dictionary 102 stores a plurality of words and word vectors of the plurality of words. When constructing the data dictionary 102, for each of the plurality of words in the data dictionary 102, the definition and the Chinese and English names of each word can be input. The processing circuit 104 can perform a word embedding operation on the definition and the Chinese and English names of each word to generate a word vector for each word, and store the word vector of each word in the data dictionary 102 for subsequent use. In this way, the user only needs to edit the words in the interface of the data dictionary without manually setting the rules of synonyms, related words, etc. in a time-consuming manner. For example, the processing circuit 104 can use a natural language processing model to perform an embedding operation on each word to generate a word vector for each word. For example, the natural language processing model may include the text-embedding-ada-002 embedding model of the OPENAI program software or the bidirectional encoder representations from transformers (BERT) model, but is not limited thereto.
[0018] Please refer to Figure 2 , Figure 2 FIG. 2 is a schematic diagram of a process 20 according to an embodiment of the present invention. The process 20 includes the following steps:
[0019] Step S200: Start.
[0020] Step S202: Obtain an input word and perform a word embedding operation on the input word to generate a word vector of the input word.
[0021] Step S204: Calculate the degree of association between the input vocabulary and a plurality of vocabulary in the data dictionary according to the word vector of the input vocabulary.
[0022] Step S206: Determine at least one recommended synonym from the plurality of words in the data dictionary according to the degree of association between the input word and the plurality of words in the data dictionary.
[0023] Step S208: End.
[0024] According to process 20, in step S202, when the user has a need for synonym search, an input word can be input to the data center system 1. After the processing circuit 104 receives the input word input by the input user, it can perform a word embedding operation on the input word to generate a word vector of the input word. For example, the processing circuit 104 can use a natural language processing model to perform an embedding operation on the input word to generate a word vector of the input word. The processing circuit 104 can store the input word and the word vector of the input word in the data dictionary 102 for subsequent use. Figure 3 As shown, after the vocabulary is converted into a word vector, the word vector can be represented by a dot in three-dimensional space. Each dot represents a vocabulary (such as an input vocabulary or a vocabulary in the data dictionary 102), wherein the vocabulary dots in related fields will be close to each other. The processing circuit 104 calculates the word vectors of the input vocabulary and the plural vocabulary of the data dictionary 102. The word vector can be the distribution distance of the input vocabulary and the plural vocabulary of the data dictionary 102 in space. Through the distribution of the word vector, a weak connection can be established between the input vocabulary and the plural vocabulary of the data dictionary 102.
[0025] In step S204, the processing circuit 104 may calculate the association degree between the input word and the plurality of words in the data dictionary 102 according to the word vector of the input word. For example, the processing circuit 104 may calculate a cosine similarity between a word vector of the input word and each of the word vectors of the plurality of words as the association degree between the input word and the word in the data dictionary 102.
[0026] In step S206, the processing circuit 104 may determine at least one recommended synonym from the plurality of words in the data dictionary 102 according to the degree of association between the input word and the plurality of words in the data dictionary 102. For example, the processing circuit 104 may determine at least one word in the plurality of words in the data dictionary 102 that is highly correlated with the input word according to the degree of association between the input word and the plurality of words in the data dictionary 102, and determine at least one word that is highly correlated with the input word as a recommended synonym. And the processing circuit 104 may output the recommended synonyms to provide to the user. For example, assuming that 991 words are collected in the data dictionary 102 in the current data governance process, and the input word that the user wants to enter to search in step S202 is "revenue related information", after processing in steps S204 and S206, the processing circuit 104 outputs the following Figure 4 The recommended synonyms shown are provided to the user.
[0027] In step S206, the processing circuit 104 may determine whether the cosine similarity between the word vector of the input vocabulary and the word vector of each vocabulary in the data dictionary 102 is greater than a threshold value. When it is determined that the cosine similarity between the word vector of the input vocabulary and the word vector of a vocabulary among the multiple vocabulary in the data dictionary 102 is greater than the threshold value, the processing circuit 104 may determine the vocabulary in the data dictionary 102 as a recommended synonym. In this way, the embodiment of the present invention can determine the recommended synonyms for users to use by calculating the degree of association through the word vectors of the vocabulary, without the need to manually set the synonyms of each vocabulary. As the company continues to promote data governance, the vocabulary library will become larger and larger, and the method of recommending synonyms using the embodiment of the present invention will effectively improve the efficiency of time and manpower.
[0028] In step S206, after the recommended synonyms are determined, the user can review the determined recommended synonyms and decide whether to further set the recommended synonyms as synonyms of the input word. After deciding to set the recommended synonyms as synonyms of the input word, the synonyms of the input word can be stored in the data dictionary 102. Later, when a user inputs a search word item that is the input word, the data dictionary 102 can also output the set synonym information of the input word for the user's reference.
[0029] A person with ordinary knowledge in the art may combine, modify or change the above-described embodiments according to the spirit of the present invention, but is not limited thereto. All of the above statements, steps, and / or processes (including recommended steps) can be implemented by hardware, software, firmware (i.e., a combination of hardware devices and computer instructions, the data in the hardware devices are read-only software data), electronic systems, or a combination of the above devices. The device may be a data center system 1. The hardware may include analog, digital and mixed circuits (i.e., microcircuits, microchips or silicon chips). For example, the hardware may be an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic component, a coupled hardware component, or a combination of the above hardware. In other embodiments, the hardware may include a general-purpose processor, a microprocessor, a controller, a digital signal processor (DSP), or a combination of the above hardware. The software may be a combination of program codes, a combination of instructions and / or a combination of functions (functionality), which is stored in a storage device, such as a computer-readable recording medium or a non-transitory computer-readable medium. For example, a computer-readable recording medium may include a read-only memory (ROM), a flash memory (Flash Memory), a random-access memory (RAM), a subscriber identity module (SIM), a hard disk, a floppy disk, or a CD-ROM (CD-ROM / DVD-ROM / BD-ROM), but is not limited thereto. The data center system 1 of an embodiment of the present invention may include a processing circuit 104 and a storage device. The process steps and embodiments of the present invention may be compiled into a program code or instruction form and stored in the storage device. The processing circuit 104 can be used to read and execute the program code or instructions stored in the storage device to implement all the aforementioned steps and functions.
[0030] In summary, the simple and fast method of the embodiment of the present invention can determine the recommended synonyms for users by calculating the degree of association through the word vector of the vocabulary, without manually setting the synonyms of each vocabulary. As the company continues to promote data governance, the vocabulary library will become larger and larger. The method of recommending synonyms using the embodiment of the present invention will effectively improve the efficiency of time and manpower.
[0031] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.
Claims
1. A data search method for a data dictionary, characterized in that: include: Obtaining an input word and performing a word embedding operation on the input word to generate a word vector of the input word; According to the word vector of the input vocabulary, the association degree between the input vocabulary and a plurality of vocabulary of the data dictionary is calculated, wherein the steps include: Calculating a cosine similarity between the word vector of the input vocabulary and each of the plurality of word vectors of the plurality of vocabulary; and At least one recommended synonym is determined from the plurality of words in the data dictionary according to the degree of association between the input word and the plurality of words in the data dictionary, wherein the step comprises: when it is determined that the cosine similarity between the word vector of the input word and a word vector of a first word in the plurality of words in the data dictionary is greater than a threshold value, determining the first word as the recommended synonym.
2. The data search method according to claim 1, characterized in that: The step of performing the word embedding operation on the input vocabulary to generate the word vector of the input vocabulary comprises: A natural language processing model is used to perform the embedding operation on the input vocabulary to generate the vector of the input vocabulary.
3. The data search method according to claim 1, characterized in that: Also includes: A natural language processing model is used to perform the embedding operation on a plurality of words in the data dictionary to generate word vectors of the plurality of words.
4. The data search method according to claim 1, characterized in that: The step of determining at least one recommended synonym from the plurality of words in the data dictionary according to the degree of association between the input word and the plurality of words in the data dictionary comprises: According to the association degree between the input word and the plurality of words in the data dictionary, at least one word in the plurality of words in the data dictionary that is highly related to the input word is determined, and the at least one word that is highly related to the input word is determined as the at least one recommended synonym.
5. A data center system, characterized in that: include: a data dictionary, including plural words; a processing circuit coupled to the data dictionary, for obtaining an input word and performing a word embedding operation on the input word to generate a word vector of the input word; Wherein, the processing circuit calculates the degree of association between the input vocabulary and multiple vocabulary in the data dictionary based on the word vector of the input vocabulary, and determines at least one recommended synonym from the multiple vocabulary in the data dictionary based on the degree of association between the input vocabulary and the multiple vocabulary in the data dictionary, wherein the processing circuit calculates a cosine similarity between the word vector of the input vocabulary and each of the multiple word vectors of the multiple vocabulary, and when it is determined that the cosine similarity between the word vector of the input vocabulary and a word vector of a first vocabulary in the multiple vocabulary of the data dictionary is greater than a threshold value, the processing circuit determines the first vocabulary as the recommended synonym.
6. The data center system according to claim 5, characterized in that: The processing circuit performs the embedding operation on the input vocabulary using a natural language processing model to generate the vector of the input vocabulary.
7. The data center system according to claim 5, characterized in that: The processing circuit uses a natural language processing model to perform the embedding operation on a plurality of words in the data dictionary to generate word vectors of the plurality of words.
8. The data center system according to claim 5, characterized in that: The processing circuit determines at least one word in the data dictionary that is highly correlated with the input word according to the degree of association between the input word and the plurality of words in the data dictionary, and determines the at least one word that is highly correlated with the input word as the at least one recommended synonym.