Word discovery method, electronic equipment and storage medium

Abstract

The invention provides a word discovery method, electronic equipment and a storage medium, and belongs to the technical field of information processing.The method comprises the steps that original text corpora in a target field are obtained, and the target field is a vertical field of new words to be recognized; determining a first word vector matrix based on the original text corpus, wherein the first word vector matrix is a vector matrix of vocabularies in a vocabulary set different from the general vocabularies in the original text corpus; clustering segmented words in the candidate vocabulary set based on the first word vector matrix to obtain a plurality of segmented word clusters; and forming a target lexicon of the target field by the word segmentation clusters belonging to the target field in the plurality of word segmentation clusters, wherein the target lexicon is used for providing vocabulary resources for the target field. Through the scheme, words in the target field are screened, and the accuracy of screening new words in the target field is improved.

Description

Technical Field

[0001] The present application belongs to the field of information processing technology, and in particular relates to a word discovery method, an electronic device and a storage medium. Background Art

[0002] With the diversification of information processing methods and the convenience of information dissemination, new information in various vertical fields is generated faster and faster, and new words in various industries emerge in an endless stream. In the fields of text processing and information mining, it has become a major problem to quickly and accurately identify new words in the industry. Usually, the identification of new words can be achieved by discovering new words in the corpus and then forming a vocabulary set with the new words.

[0003] At present, new words are determined by information entropy. The information entropy threshold is determined based on the experience of industry personnel and relevant factors such as corpus size and distribution, and then new words in the corpus are determined based on the information entropy threshold.

[0004] In the above-mentioned word discovery process, affected by relevant factors such as the size and distribution of the corpus, the method of determining new words through information entropy threshold results in more new word impurities in the vertical field, and the granularity of the new words is too large or too small, resulting in inaccurate new word recognition. Summary of the invention

[0005] The purpose of the present application is to provide a word discovery method, an electronic device and a storage medium, aiming to solve the problems of low efficiency and high labor cost in the traditional word discovery process.

[0006] A first aspect of an embodiment of the present application provides a word discovery method, the method comprising:

[0007] Acquire original text corpus in a target domain, where the target domain is a vertical domain of new words to be identified;

[0008] Determine a first word vector matrix based on the original text corpus, where the first word vector matrix is ​​a vector matrix of words in the candidate vocabulary set that are different from common words in the original text corpus;

[0009] Clustering the words in the candidate vocabulary set based on the first word vector matrix to obtain multiple word clusters;

[0010] The word segmentation clusters belonging to the target domain among the multiple word segmentation clusters are combined into a target word library of the target domain, and the target word library is used to provide vocabulary resources for the target domain.

[0011] In some embodiments, determining a first word vector matrix based on the original text corpus includes:

[0012] Performing word segmentation and parsing on the original text corpus to obtain an initial vocabulary set corresponding to the original text corpus;

[0013] Acquire a preset vocabulary set, wherein the preset vocabulary set is a general vocabulary set;

[0014] Deleting the word segments in the preset vocabulary set from the initial vocabulary set to obtain a candidate vocabulary set corresponding to the original text corpus;

[0015] The word vectors of each word in the candidate vocabulary set are combined into the first word vector matrix.

[0016] In some embodiments, forming the first word vector matrix from the word vectors of each word segment in the candidate vocabulary set includes:

[0017] Determining a first word vector for each word segment in the initial vocabulary set; and determining a second word vector for each word segment in the preset vocabulary set;

[0018] Deleting the word vector that is the same as the second word vector from the plurality of first word vectors, to obtain a first word vector corresponding to each word in the candidate vocabulary set;

[0019] The first word vector matrix is ​​formed by combining the first word vector corresponding to each word segment in the candidate vocabulary set.

[0020] In some embodiments, forming the first word vector matrix from the first word vector corresponding to each word segment in the candidate vocabulary set includes:

[0021] Counting the semantic weight of each word in the initial vocabulary set, wherein the semantic weight indicates the importance of the word in the original text corpus;

[0022] amplifying the feature difference between the first word vectors corresponding to each word in the candidate vocabulary set based on the semantic weight of each word in the initial vocabulary set;

[0023] The enlarged multiple first word vectors are combined into the first word feature matrix.

[0024] In some embodiments, clustering the words in the candidate vocabulary set based on the first word vector matrix to obtain a plurality of word clusters includes:

[0025] Performing dimensionality reduction processing on the first word vector matrix to obtain a second word vector matrix;

[0026] The word segments in the candidate vocabulary set are clustered based on the second word vector matrix to obtain multiple word segmentation clusters.

[0027] In some embodiments, the step of performing dimensionality reduction processing on the first word vector matrix to obtain a second word vector matrix includes:

[0028] Determine multiple singular values ​​of the first word vector matrix and left and right singular vectors corresponding to each singular value;

[0029] Sorting the plurality of singular values ​​from large to small to obtain a singular value sequence consisting of the plurality of singular values;

[0030] Determine a first target number of target singular values ​​from the singular value sequence;

[0031] The inverse vectors of the left and right singular vectors corresponding to the target number of target singular values ​​are used to form the second word vector matrix.

[0032] In some embodiments, before clustering the words in the candidate vocabulary set based on the second word vector matrix to obtain a plurality of word clusters, the method further includes:

[0033] Determine the difference between the first word vector matrix and the second word vector matrix to obtain a third word vector matrix;

[0034] Amplify the difference between each vector element in the third word vector matrix.

[0035] In some embodiments, clustering the words in the candidate vocabulary set based on the first word vector matrix to obtain a plurality of word clusters includes:

[0036] Determine similarity threshold;

[0037] Clustering the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result;

[0038] Amplifying the similarity threshold based on a preset incremental strategy, and based on the amplified similarity threshold, continuing to perform the step of clustering the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result, and updating the clustering result;

[0039] Until the clustering is completed, multiple word clusters are obtained.

[0040] A second aspect of an embodiment of the present application provides a word discovery device, the device comprising:

[0041] An acquisition unit, used to acquire original text corpus in a target domain, wherein the target domain is a vertical domain of new words to be identified;

[0042] A first determining unit is used to determine a first word vector matrix based on the original text corpus, where the first word vector matrix is ​​a vector matrix of words in the candidate vocabulary set that are different from common words in the original text corpus;

[0043] A clustering unit, configured to cluster the words in the candidate vocabulary set based on the first word vector matrix to obtain a plurality of word clusters;

[0044] The forming unit is used to form a target word library of the target field by combining the word clusters belonging to the target field among the multiple word clusters, and the target word library is used to provide vocabulary resources for the target field.

[0045] In some embodiments, the first determination unit is used to perform word segmentation and parsing on the original text corpus to obtain an initial vocabulary set corresponding to the original text corpus; obtain a preset vocabulary set, which is a general vocabulary set; delete the word segments existing in the preset vocabulary set from the initial vocabulary set to obtain a candidate vocabulary set corresponding to the original text corpus; and form the word vector of each word segment in the candidate vocabulary set into the first word vector matrix.

[0046] In some embodiments, the first determination unit is used to determine a first word vector for each word segment in the initial vocabulary set; and determine a second word vector for each word segment in the preset vocabulary set; delete the word vector that is identical to the second word vector from multiple first word vectors to obtain a first word vector corresponding to each word segment in the candidate vocabulary set; and form the first word vector corresponding to each word segment in the candidate vocabulary set into the first word vector matrix.

[0047] In some embodiments, the first determination unit is used to count the semantic weight of each word segment in the initial vocabulary set, where the semantic weight represents the importance of the word segment in the original text corpus; amplify the feature difference between the first word vectors corresponding to each word segment in the candidate vocabulary set based on the semantic weight of each word segment in the initial vocabulary set; and compose the amplified multiple first word vectors into the first word feature matrix.

[0048] In some embodiments, the clustering unit is used to perform dimensionality reduction processing on the first word vector matrix to obtain a second word vector matrix; and cluster the word segments in the candidate vocabulary set based on the second word vector matrix to obtain multiple word segment clusters.

[0049] In some embodiments, the clustering unit is used to determine multiple singular values ​​of the first word vector matrix and the left and right singular vectors corresponding to each singular value; sort the multiple singular values ​​from large to small to obtain a singular value sequence composed of the multiple singular values; determine the first target number of target singular values ​​from the singular value sequence; and form the second word vector matrix with the inverse vectors of the left and right singular vectors corresponding to the target number of target singular values.

[0050] In some embodiments, the apparatus further comprises:

[0051] A second determining unit is used to determine the difference between the first word vector matrix and the second word vector matrix to obtain a third word vector matrix;

[0052] A difference amplification unit is used to amplify the difference between each vector element in the third word vector matrix.

[0053] In some embodiments, the clustering unit is used to determine a similarity threshold; cluster the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result; amplify the similarity threshold based on a preset incremental strategy, and continue to perform the step of clustering the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result based on the amplified similarity threshold, and update the clustering result; until clustering is completed and multiple word clusters are obtained.

[0054] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned word discovery method when executing the computer program.

[0055] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the word discovery method described above is implemented.

[0056] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0057] In an embodiment of the present application, a first word vector matrix corresponding to words in a candidate vocabulary set that is different from general words in the original text corpus in the vertical field is first screened out, and then the word vectors in the word vector matrix are clustered to obtain word clusters of multiple categories, and then the word clusters belonging to the target field are screened from the multiple word clusters, and the word clusters belonging to the target field are organized into a target vocabulary of the target field, thereby providing vocabulary resources for the target field. In this way, candidate new words that do not exist in the general vocabulary are first screened out from the original text corpus, so that words that may be new words in the target field are preliminarily screened out, reducing the amount of calculation during clustering, and determining the first word vector matrix corresponding to the words in the candidate vocabulary set. The segmented words in the candidate vocabulary set are clustered based on the first word vector matrix, and then new words belonging to the target field are screened from the clustering results to form a target vocabulary. In this way, words with similar word vectors are clustered into a group through clustering, thereby dividing the words in the candidate vocabulary set into multiple component word clusters, so that the segmented word clusters belonging to the target field can be screened out from the multiple component word clusters. In this way, words that do not belong to the target field are deleted through clustering, and the words in the target field are further screened, thereby improving the accuracy of screening new words in the target field. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram showing a flow chart of a word discovery method provided by an exemplary embodiment;

[0059] Figure 2 A schematic diagram showing a flow chart of a word discovery method provided by an exemplary embodiment;

[0060] Figure 3 A schematic diagram showing an initial vocabulary set provided by an exemplary embodiment is shown;

[0061] Figure 4 A schematic diagram showing a corresponding relationship between each word segment in a candidate vocabulary set and a first word vector M3 provided by an exemplary embodiment;

[0062] Figure 5 A schematic diagram showing the semantic weight corresponding to each word segment in the initial vocabulary set provided by an exemplary embodiment is shown;

[0063] Figure 6 A schematic diagram showing a corresponding relationship between each word segment in the initial vocabulary set and the first word vector M2 after amplifying the feature difference between the first word vectors in the initial vocabulary set provided by an exemplary embodiment;

[0064] Figure 7 An exemplary embodiment provides a diagram showing a plurality of first word vectors enlarged from a candidate vocabulary set to form the first word feature matrix;

[0065] Figure 8A schematic diagram showing a flow chart of a word discovery method provided by an exemplary embodiment;

[0066] Fig. 9 A schematic diagram of a word vector of a segmentation after a difference between vector elements is enlarged provided by an exemplary embodiment is shown;

[0067] Fig.10 A schematic diagram of the structure of a word discovery device provided by an exemplary embodiment is shown;

[0068] Fig.11 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0071] With the diversification of information processing methods and the convenience of information dissemination, the speed of new information generation in various vertical fields is getting faster and faster, and new words in various industries are emerging in an endless stream. In the fields of text processing and information mining, it has become a major problem to quickly and accurately identify new words in the industry. Usually, the identification of new words can be achieved by discovering new words in the corpus and then establishing a vocabulary set.

[0072] In some embodiments, the word discovery of vertical field corpus generally adopts the method of training neural network by text word annotation. In the method of training neural network by text word annotation, experts in the vertical field are required to annotate each word to obtain a training set for neural network training, and the neural network is trained based on the training set, and the new words in the recognition corpus are identified based on the trained neural network.

[0073] In the process of word discovery, a large number of training sets need to be collected and annotated. The annotation process requires experts in the field to annotate word by word, which is time-consuming and labor-intensive, inefficient, and has high labor costs.

[0074] In other embodiments, new words are determined by information entropy, and an information entropy threshold is determined based on the experience of industry personnel and relevant factors such as the size and distribution of the corpus. Then, new words in the corpus are determined based on the information entropy threshold.

[0075] In the above-mentioned word discovery process, affected by relevant factors such as the size and distribution of the corpus, the method of determining new words through information entropy threshold results in more new word impurities in the vertical field, and the granularity of the new words is too large or too small, resulting in inaccurate new word recognition.

[0076] The present application provides a word discovery method, electronic device and storage medium. First, the first word vector matrix corresponding to the vocabulary in the candidate vocabulary set that is different from the general vocabulary in the original text corpus in the vertical field is screened out, and then the word vectors in the word vector matrix are clustered to obtain multiple categories of word clusters, and then the word clusters belonging to the target field are screened from the multiple word clusters, and the word clusters belonging to the target field are formed into a target vocabulary of the target field to provide vocabulary resources for the target field. In this way, the vocabulary in the candidate vocabulary set in the original text corpus is first screened according to the general vocabulary, and the first word vector matrix corresponding to the vocabulary in the candidate vocabulary set is determined, the word segments in the candidate vocabulary set are clustered based on the first word vector matrix, and then the new words belonging to the target field are screened from the clustering results to form a target vocabulary, so that the new words in the target field are determined after multiple screenings, avoiding the training process of a large neural network model, and then avoiding manual labeling of the training set, saving manpower and material resources.

[0077] The terms involved in this application are explained below.

[0078] Vertical fields: Compared with daily general fields, vertical fields refer to market segments that conduct in-depth research and exploration in a specific industry or field, emphasizing professionalism, depth and uniqueness. For example, finance, medical care, law, transportation, etc. can all become vertical fields.

[0079] Word discovery: It can also be called "new mining", which refers to the technology of finding words that are unique to a vertical field from the text corpus of a vertical field.

[0080] Statistical language model: A mathematical model that solves the context-related characteristics of natural language from the perspective of probability statistics. The core of the statistical language model is to determine the probability of a word appearing in a text.

[0081] TF-IDF: Term Frequency-Inverse Document Frequency, is a statistical method used to evaluate the importance of a word to a document set or a document in a corpus.

[0082] Word Vector: Word vector is a general term for a set of language modeling and feature learning techniques in word embedded natural language processing (NLP), which maps words or phrases from the vocabulary to vectors of real numbers. Conceptually, it designs mathematical embedding from a one-dimensional space for each word to a continuous vector space with lower dimensions.

[0083] Singular value decomposition: refers to decomposing an arbitrary matrix into three matrices. That is, it can be expressed as:

[0084] in, Accordingly,

[0086] Among them, A is an m×n matrix, U is the left singular vector of A, which is an m×m orthogonal matrix, S is the singular value of A, which is an m×n non-negative diagonal matrix, and V is the right singular vector of A, which is an n×n orthogonal matrix.

[0087] The present application is described below in conjunction with specific embodiments. Figure 1 , which shows a flowchart of a new word recognition method provided by an exemplary embodiment. As an example but not a limitation, the method is applied to an electronic device with a data processing function. The electronic device may be a personal computer, a mobile terminal (such as a mobile phone, a tablet computer), a wearable device (such as a smart watch), or a smart car (such as a vehicle equipped with a smart cockpit system).

[0088] S101, the electronic device obtains original text corpus in a target field, where the target field is a vertical field of new words to be recognized.

[0089] The original text corpus is related public text data or audio data in the target field. When the original text corpus is audio data, the audio data can be converted into text data through text conversion. The electronic device can obtain the original text corpus from a public database in the target field. For example, the electronic device obtains the original text corpus from a public database such as a forum or question post in the target field. Alternatively, the electronic device can obtain the original text corpus associated with the target field from the database of the voice assistant.

[0090] One point that needs to be explained is that when acquiring the original text corpus related to the user, the electronic device sends an acquisition request to the user, and only when a confirmation operation for the acquisition request is received does the electronic device execute the step of acquiring the original text corpus.

[0091] S102, the electronic device determines a first word vector matrix based on the original text corpus, where the first word vector matrix is ​​a vector matrix of words in the candidate vocabulary set that are different from common words in the original text corpus.

[0092] The common vocabulary is a vocabulary determined according to a common vocabulary set. The common vocabulary set is an existing vocabulary set, for example, the common vocabulary set is a jieba vocabulary library, etc. In the embodiment of the present application, the common vocabulary set is not specifically limited.

[0093] In this step, the electronic device performs word segmentation analysis, word vector extraction and other processing on the collected original text corpus to obtain multiple word segments in the original text corpus, and then extracts the word vectors of the multiple word segments, and compares the multiple word segments of the original text corpus with common vocabulary, screens out vocabulary in the candidate vocabulary set that is different from the common vocabulary in the original text corpus, determines the word vector of the vocabulary in the candidate vocabulary set from the multiple word vectors, and forms the first word vector matrix based on the word vectors of the multiple candidate new words.

[0094] S103: The electronic device clusters the word segments in the candidate vocabulary set based on the first word vector matrix to obtain a plurality of word segment clusters.

[0095] In this step, the electronic device performs similarity clustering on the word segments in the candidate vocabulary set based on the first word vector matrix to obtain a plurality of word segment clusters corresponding to the vocabulary set.

[0096] In some embodiments, the electronic device clusters the words in the first word vector matrix based on increasing similarity. This process can be implemented by the following steps S1031-S1034, including:

[0097] S1031, the electronic device determines a similarity threshold.

[0098] When similarity clustering is started, an initial similarity threshold may be set, which may be set based on the experience of a person skilled in the art, and is not specifically limited in the embodiments of the present application. For example, the initial similarity threshold may be 0.6, 0.65, or 0.67, etc.

[0099] S1032: The electronic device clusters the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result.

[0100] In this step, the electronic device performs similarity clustering on the word vectors of each word segment in the first word vector matrix based on a preset clustering algorithm. The preset algorithm may be any similarity clustering algorithm, such as a hierarchical clustering algorithm. The electronic device clusters the word segments whose word vectors meet the similarity threshold into one category through the similarity threshold. For example, word segment A is similar to word segment B, and word segment B is similar to word segment C, then word segment A, word segment B, and word segment C are clustered into one category.

[0101] The method for determining the similarity of words by the electronic device can be set as needed. In the embodiment of the present application, the method for determining the similarity of words is not specifically limited. For example, the electronic device can determine the similarity between words based on the Euclidean distance, Mahalanobis distance, etc. between the word vectors corresponding to the words.

[0102] S1033, the electronic device amplifies the similarity threshold based on a preset incremental strategy, and based on the amplified similarity threshold, continues to execute the step of clustering the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result, and updates the clustering result.

[0103] The preset incremental strategy can be set as needed, and in the embodiment of the present application, the preset incremental strategy is not specifically limited. For example, the preset incremental strategy can be a power exponential incremental strategy or a logarithmic incremental strategy.

[0104] The electronic device amplifies the similarity threshold based on a preset incremental strategy, and then clusters the candidate words in the first word vector matrix based on the increased similarity threshold, and thus loops through steps S1031-S1033, iterates clustering of multiple words, and obtains a clustering result.

[0105] S1034, until clustering is completed, multiple word clusters are obtained.

[0106] In this implementation, the vocabulary is clustered by using an iterative clustering method with an increased similarity threshold, thereby preventing all vocabulary from being clustered into one category and improving the accuracy of vocabulary clustering.

[0107] S104, the electronic device combines the word segmentation clusters belonging to the target domain among the multiple word segmentation clusters into a target word library of the target domain, and the target word library is used to provide vocabulary resources for the target domain.

[0108] In this step, the multiple word segmentation clusters are screened, and the word segmentation clusters of the target field are screened out from the multiple word segmentation clusters, and the word segmentation clusters of the target field are combined into a target word library of the target field. The electronic device writes the words corresponding to different clusters into different files, and the different files form the target word library of the target field; or the electronic device writes the words corresponding to different clusters into one file to obtain new words in the target field.

[0109] In some embodiments, before generating a new vocabulary, the electronic device may also filter the words in each word cluster using preset filtering rules, and delete adverbs, modal particles, affixes, etc. in the word cluster to make the final target vocabulary more accurate.

[0110] After the electronic device generates the target vocabulary, it can store the target vocabulary locally for local application calls. The electronic device can also send the target vocabulary to other electronic devices or cloud storage. Accordingly, other electronic devices can obtain the target vocabulary from the cloud storage.

[0111] Among them, the cloud storage can establish communication connections with multiple electronic devices, and the target word library can also carry the field identification of the corresponding target field. When the cloud storage receives the target word library sent by multiple electronic devices, the cloud storage determines the field identification corresponding to each target word library. When there are target words with the same field identification, the cloud storage compares the word content in the multiple target words with the same field identification, and stores different words in the multiple target words as the word library corresponding to the target field. In this way, the word library corresponding to the target field is determined by different electronic devices, making the vocabulary in the word library richer.

[0112] In an embodiment of the present application, a first word vector matrix corresponding to words in a candidate vocabulary set that is different from general words in the original text corpus in the vertical field is first screened out, and then the word vectors in the word vector matrix are clustered to obtain word clusters of multiple categories, and then the word clusters belonging to the target field are screened from the multiple word clusters, and the word clusters belonging to the target field are organized into a target vocabulary of the target field, thereby providing vocabulary resources for the target field. In this way, candidate new words that do not exist in the general vocabulary are first screened out from the original text corpus, so that words that may be new words in the target field are preliminarily screened out, reducing the amount of calculation during clustering, and determining the first word vector matrix corresponding to the words in the candidate vocabulary set. The segmented words in the candidate vocabulary set are clustered based on the first word vector matrix, and then new words belonging to the target field are screened from the clustering results to form a target vocabulary. In this way, words with similar word vectors are clustered into a group through clustering, thereby dividing the words in the candidate vocabulary set into multiple component word clusters, so that the segmented word clusters belonging to the target field can be screened out from the multiple component word clusters. In this way, words that do not belong to the target field are deleted through clustering, and the words in the target field are further screened, thereby improving the accuracy of screening new words in the target field.

[0113] In practical applications, when determining the first word vector matrix of the original text corpus in the above step 102, the electronic device can first determine the difference between the original text corpus and the preset vocabulary set to obtain a candidate vocabulary set, and then determine the first word vector matrix corresponding to the word segmentation in the candidate vocabulary set. Combined with the above process of determining the first word vector matrix, the present application also provides a method for word discovery, see Figure 2 , which shows a flow chart of a word discovery method provided by an exemplary embodiment. As an example but not a limitation, the method is applied to an electronic device.

[0114] S201, the electronic device obtains original text corpus in a target field, where the target field is a vertical field of new words to be recognized.

[0115] The principle of this step is the same as that of step S101 and will not be repeated here.

[0116] S202, the electronic device performs word segmentation analysis on the original text corpus to obtain an initial vocabulary set corresponding to the original text corpus.

[0117] In some embodiments, the electronic device determines the initial vocabulary set corresponding to the original text corpus through a preset language statistical model. Accordingly, the electronic device calculates the statistical language model M1 based on the original text corpus to obtain the initial vocabulary set S0. Figure 3 , which shows an initial vocabulary set corresponding to the original text corpus provided by an exemplary embodiment. The initial vocabulary set includes multiple word segments and a sequence number or index corresponding to each word segment. Figure 3As shown, the initial vocabulary set includes words such as "of", "place name", "bluetooth", "connection", etc.; and the sequence numbers corresponding to the multiple word segments are counted from 0 to n.

[0118] S203: The electronic device obtains a preset vocabulary set, which is a general vocabulary set.

[0119] The electronic device may obtain the preset vocabulary set from a publicly available database, or the electronic device may also receive the preset vocabulary set uploaded by the user. In the embodiment of the present application, the manner in which the electronic device obtains the preset vocabulary set is not specifically limited.

[0120] It should be noted that the preset vocabulary set may be a general vocabulary set, for example, the preset vocabulary set is the Jieba vocabulary library.

[0121] S204, the electronic device deletes the segmented words in the preset vocabulary set from the initial vocabulary set to obtain a candidate vocabulary set corresponding to the original text corpus.

[0122] The initial vocabulary set obtained in step S202 is subtracted from the preset vocabulary set to obtain the remaining candidate vocabulary set.

[0123] S205, the electronic device combines the word vectors of each word in the candidate vocabulary set into the first word vector matrix.

[0124] In this step, the electronic device forms the first vector matrix with the word vectors of each word segment in the candidate vocabulary set.

[0125] In some embodiments, the electronic device determines the difference between the two word vector matrices based on the word vector matrix of the vocabulary in the general vocabulary set and the word vector matrix of each word in the initial vocabulary set, and obtains the first word vector matrix corresponding to the candidate vocabulary set. This process can be implemented by the following steps S2051-S2053, including:

[0126] S2051, the electronic device determines a first word vector for each word segment in the initial vocabulary set; and determines a second word vector for each word segment in the preset vocabulary set.

[0127] The electronic device trains a word vector model (Word Vector) according to the word segments in the initial vocabulary set to obtain a first word vector corresponding to each word segment in the initial vocabulary set.

[0128] Among them, the dimension of the first word vector is generally set between 64-256, and is generally an integer of 2. The dimension of the first word vector can be set as needed. In the embodiment of the present application, the dimension of the first word vector is not specifically limited.

[0129] For the second word vector of each word segment in the preset vocabulary set, the electronic device can determine the second word vector according to the same principle as determining the first word vector. Alternatively, in some embodiments, the preset vocabulary set includes multiple words and word vectors corresponding to each word, and accordingly, the electronic device can determine the word vector of each word in the preset vocabulary set as the second word vector, which is not specifically limited in the embodiments of the present application.

[0130] S2052, the electronic device deletes the word vector that is the same as the second word vector from the multiple first word vectors, and obtains the first word vector corresponding to each word in the candidate vocabulary set.

[0131] In this step, the electronic device subtracts the second word vector from the multiple first word vectors to obtain the first word vector corresponding to each word in the remaining candidate vocabulary set.

[0132] See also Figure 4 , which shows the corresponding relationship between each word segment in the candidate vocabulary set and the first word vector M1 provided by an exemplary embodiment.

[0133] S2053, the electronic device combines the first word vectors corresponding to each word segment in the candidate vocabulary set into a first word vector matrix.

[0134] In some embodiments, the electronic device may directly form the first word vector matrix with the first word vector corresponding to each acne in the candidate vocabulary. In other embodiments, the electronic device may also first determine the semantic weight of each word segment, and determine the first word vector matrix according to the semantic weight of each word segment. Accordingly, the process includes the following steps (1)-(3):

[0135] (1) The electronic device counts the semantic weight of each word in the initial vocabulary, and the semantic weight indicates the importance of the word in the original text corpus.

[0136] The semantic weight may be any parameter that can represent the importance of a word segment in the original text corpus. For example, the semantic weight may be the TF-IDF value of the word segment.

[0137] See also Figure 5 , which shows the semantic weight corresponding to each word segment in an initial vocabulary set provided by an exemplary embodiment. The vocabulary set can be represented as S1.

[0138] (2) The electronic device amplifies the feature difference between the first word vectors corresponding to each word in the candidate vocabulary set based on the semantic weight of each word in the initial vocabulary set.

[0139] In some embodiments, the electronic device uses the semantic weight of each word segment as a dimension of the word vector to increase the feature difference between the first word vectors corresponding to each word segment. Figure 6 , which shows the correspondence between each word segment in the initial vocabulary set and the first word vector M2 after amplifying the feature difference between the first word vectors in the initial vocabulary set, provided by an exemplary embodiment.

[0140] Then, the first word vector corresponding to each word in the candidate vocabulary set is determined from the first word vector corresponding to each word in the initial vocabulary set, and the first word vector corresponding to each word in the candidate vocabulary set is obtained after the feature difference is amplified.

[0141] One point that needs to be explained is that the process of the electronic device determining the semantic weight of each word in the initial vocabulary set can be processed at any time after determining the initial vocabulary set and before determining the first word feature matrix. In the embodiment of the present application, the execution order of this step is not specifically limited.

[0142] (3) The electronic device combines the amplified multiple first word vectors into the first word feature matrix.

[0143] See also Figure 7 , which shows that a plurality of first word vectors after enlargement in a candidate vocabulary set provided by an exemplary embodiment are composed into the first word feature matrix.

[0144] S206: The electronic device clusters the word segments in the candidate vocabulary set based on the first word vector matrix to obtain a plurality of word segment clusters.

[0145] The principle of this step is the same as that of step S103 and will not be repeated here.

[0146] S207, the electronic device combines the word segmentation clusters belonging to the target domain among the multiple word segmentation clusters into a target word library for the target domain, and the target word library is used to provide vocabulary resources for the target domain.

[0147] The principle of this step is the same as that of step S104 and will not be repeated here.

[0148] In an embodiment of the present application, a first word vector matrix corresponding to words in a candidate vocabulary set that is different from general words in the original text corpus in the vertical field is first screened out, and then the word vectors in the word vector matrix are clustered to obtain word clusters of multiple categories, and then the word clusters belonging to the target field are screened from the multiple word clusters, and the word clusters belonging to the target field are organized into a target vocabulary of the target field, thereby providing vocabulary resources for the target field. In this way, candidate new words that do not exist in the general vocabulary are first screened out from the original text corpus, so that words that may be new words in the target field are preliminarily screened out, reducing the amount of calculation during clustering, and determining the first word vector matrix corresponding to the words in the candidate vocabulary set. The segmented words in the candidate vocabulary set are clustered based on the first word vector matrix, and then new words belonging to the target field are screened from the clustering results to form a target vocabulary. In this way, words with similar word vectors are clustered into a group through clustering, thereby dividing the words in the candidate vocabulary set into multiple component word clusters, so that the segmented word clusters belonging to the target field can be screened out from the multiple component word clusters. In this way, words that do not belong to the target field are deleted through clustering, and the words in the target field are further screened, thereby improving the accuracy of screening new words in the target field.

[0149] Furthermore, the differences between the first word vectors of multiple word segments in the candidate vocabulary set are amplified by semantic weights to obtain a first word vector matrix, thereby amplifying the differences between word segmentation features to improve the accuracy of subsequent clustering.

[0150] In practical applications, in the above step 103, in order to simplify the calculation process, the electronic device can also first perform dimensionality reduction processing on the word vector matrix. The present application also provides a method for word discovery, see Figure 8 , which shows a flow chart of a word discovery method provided by an exemplary embodiment. As an example but not a limitation, the method is applied to an electronic device.

[0151] S801, the electronic device obtains original text corpus in a target field, where the target field is a vertical field of new words to be recognized.

[0152] The principle of this step is the same as that of step S101 and will not be repeated here.

[0153] S802, the electronic device determines a first word vector matrix based on the original text corpus, where the first word vector matrix is ​​a vector matrix of words in the candidate vocabulary set that are different from common words in the original text corpus.

[0154] The principle of this step is the same as that of step S102 and will not be repeated here.

[0155] S803, the electronic device performs dimensionality reduction processing on the first word vector matrix to obtain a second word vector matrix.

[0156] Reducing the dimension of the first word vector matrix refers to screening out the vector elements with high validity in the first word vector matrix and deleting the vector elements with low validity, so as to achieve the purpose of reducing the dimension of the first word vector matrix. In some embodiments, the electronic device can reduce the dimension of the first word vector by a singular value decomposition method. Accordingly, the process can be implemented by the following steps S8031-S8034, including:

[0157] S8031, the electronic device determines multiple singular values ​​of the first word vector matrix and left and right singular vectors corresponding to each singular value.

[0158] In this step, singular value decomposition is performed on the d-dimensional first word vector matrix to obtain d singular values ​​and corresponding left and right singular vectors.

[0159] S8032, the electronic device sorts the multiple singular values ​​from large to small to obtain a singular value sequence composed of the multiple singular values.

[0160] S8033, the electronic device determines a target number of target singular values ​​from the singular value sequence.

[0161] The target number can be determined according to the dimension of the first word vector matrix. For example, if the dimension of the first word vector matrix is ​​d, the target number can be the smallest integer rounded up to d×a, where a is an empirical parameter set by the developer, and the value of a is not specifically limited in the embodiment of the present application. For example, a is 0.6826.

[0162] S8034, the electronic device combines the inverse vectors of the left and right singular vectors corresponding to the target number of target singular values ​​into the second word vector matrix.

[0163] In this step, the electronic device determines the left and right singular vectors corresponding to each selected singular value, determines the inverse vectors of the selected left and right singular vectors, and combines the inverse vectors of multiple left and right singular vectors into a second word vector matrix.

[0164] In this implementation, the electronic device selects effective vector parameters through singular value decomposition, which reduces the dimension of the word vector matrix and the amount of subsequent clustering calculations while ensuring the accuracy of the word vector.

[0165] S804: The electronic device clusters the word segments in the candidate vocabulary set based on the second word vector matrix to obtain a plurality of word segment clusters.

[0166] The process of clustering the electronic device based on the second word vector matrix is ​​the same as the clustering process in step S103, and will not be repeated here.

[0167] It should be noted that, since the first word vector is reduced in dimension in step S803, in order to increase the difference between each word vector in the second word vector matrix after the dimension reduction, the electronic device also amplifies the vector elements in the second word vector matrix when clustering the second word vector matrix to improve the accuracy of clustering. This process can be implemented by the following steps S8041-S8044, including:

[0168] S8041, the electronic device determines the difference between the first word vector matrix and the second word vector matrix to obtain a third word vector matrix.

[0169] Since the second word vector matrix is ​​a matrix composed of the inverse vectors of the left and right singular vectors of the valid vector elements in the first word vector matrix, the difference between the first word vector matrix and the second word vector matrix can amplify the difference between the word vectors. For example, if the first word vector matrix is ​​M4 and the second word vector matrix is ​​M4 K , then the third word vector matrix M5 is M4-M4 K .

[0170] S8042: The electronic device amplifies the difference between each vector element in the third word vector matrix.

[0171] Since the third word vector matrix M5 is the first word vector matrix M4 and the second word vector matrix M4 K Therefore, the vector element value in the third word vector matrix M5 may be very small. In order to ensure the accuracy of clustering, the electronic device increases the difference between different vector elements.

[0172] In some embodiments, the electronic device amplifies the difference between the vector elements based on the Matthew effect, for example, taking an exponential function e with the natural exponent as the base for each vector element in the third word vector matrix M5. x , thus obtaining M6, see Fig. 9 , which shows a word vector of a word segment in M6 provided by an exemplary embodiment. Wherein, x refers to the value of each vector element in the third word vector matrix M5.

[0173] In this implementation, the electronic device amplifies the vector elements in the word vector matrix after dimensionality reduction, so that the difference between the vector elements increases, thereby improving the accuracy of clustering.

[0174] S805: The electronic device groups the word clusters belonging to the target domain among the multiple word clusters into a target word library for the target domain, and the target word library is used to provide vocabulary resources for the target domain.

[0175] The principle of this step is the same as that of step S104 and will not be repeated here.

[0176] In an embodiment of the present application, a first word vector matrix corresponding to words in a candidate vocabulary set that is different from general words in the original text corpus in the vertical field is first screened out, and then the word vectors in the word vector matrix are clustered to obtain word clusters of multiple categories, and then the word clusters belonging to the target field are screened from the multiple word clusters, and the word clusters belonging to the target field are organized into a target vocabulary of the target field, thereby providing vocabulary resources for the target field. In this way, candidate new words that do not exist in the general vocabulary are first screened out from the original text corpus, so that words that may be new words in the target field are preliminarily screened out, reducing the amount of calculation during clustering, and determining the first word vector matrix corresponding to the words in the candidate vocabulary set. The segmented words in the candidate vocabulary set are clustered based on the first word vector matrix, and then new words belonging to the target field are screened from the clustering results to form a target vocabulary. In this way, words with similar word vectors are clustered into a group through clustering, thereby dividing the words in the candidate vocabulary set into multiple component word clusters, so that the segmented word clusters belonging to the target field can be screened out from the multiple component word clusters. In this way, words that do not belong to the target field are deleted through clustering, and the words in the target field are further screened, thereby improving the accuracy of screening new words in the target field.

[0177] Furthermore, before clustering the features, the dimension of the feature matrix is ​​reduced, which reduces the amount of calculation for subsequent clustering while ensuring the accuracy of the word vector.

[0178] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0179] See also Fig.10 , which shows a schematic diagram of the structure of a word discovery device provided by the present application, and includes various units for executing various steps in the above embodiments, see Fig.10 , the word finding device comprises:

[0180] An acquisition unit 1001 is used to acquire original text corpus in a target domain, where the target domain is a vertical domain of new words to be identified;

[0181] A first determining unit 1002 is used to determine a first word vector matrix based on the original text corpus, where the first word vector matrix is ​​a vector matrix of words in the candidate vocabulary set that are different from common words in the original text corpus;

[0182] A clustering unit 1003 is used to cluster the words in the candidate vocabulary set based on the first word vector matrix to obtain a plurality of word clusters;

[0183] The forming unit 1004 is used to form a target word library of the target field with the word clusters belonging to the target field among the multiple word clusters. The target word library is used to provide vocabulary resources for the target field.

[0184] In some embodiments, the first determination unit 1002 is used to perform word segmentation and parsing on the original text corpus to obtain an initial vocabulary set corresponding to the original text corpus; obtain a preset vocabulary set, which is a general vocabulary set; delete the word segments existing in the preset vocabulary set from the initial vocabulary set to obtain a candidate vocabulary set corresponding to the original text corpus; and form the word vector of each word segment in the candidate vocabulary set into the first word vector matrix.

[0185] In some embodiments, the first determination unit 1002 is used to determine the first word vector of each word segment in the initial vocabulary set; and determine the second word vector of each word segment in the preset vocabulary set; delete the word vector that is identical to the second word vector from multiple first word vectors to obtain the first word vector corresponding to each word segment in the candidate vocabulary set; and form the first word vector corresponding to each word segment in the candidate vocabulary set into the first word vector matrix.

[0186] In some embodiments, the first determination unit 1002 is used to count the semantic weight of each word segment in the initial vocabulary set, where the semantic weight represents the importance of the word segment in the original text corpus; amplify the feature difference between the first word vectors corresponding to each word segment in the candidate vocabulary set based on the semantic weight of each word segment in the initial vocabulary set; and compose the amplified multiple first word vectors into the first word feature matrix.

[0187] In some embodiments, the clustering unit 1003 is used to perform dimensionality reduction processing on the first word vector matrix to obtain a second word vector matrix; cluster the word segments in the candidate vocabulary set based on the second word vector matrix to obtain multiple word segment clusters.

[0188] In some embodiments, the clustering unit 1003 is used to determine multiple singular values ​​of the first word vector matrix and the left and right singular vectors corresponding to each singular value; sort the multiple singular values ​​from large to small to obtain a singular value sequence composed of the multiple singular values; determine a target number of target singular values ​​from the singular value sequence; and form the second word vector matrix with the inverse vectors of the left and right singular vectors corresponding to the target number of target singular values.

[0189] In some embodiments, the apparatus further comprises:

[0190] A second determining unit is used to determine the difference between the first word vector matrix and the second word vector matrix to obtain a third word vector matrix;

[0191] The difference amplification unit is used to amplify the difference between each vector element in the third word vector matrix.

[0192] In some embodiments, the clustering unit 1003 is used to determine a similarity threshold; cluster the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result; amplify the similarity threshold based on a preset incremental strategy, and continue to perform the step of clustering the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result based on the amplified similarity threshold, and update the clustering result; until the clustering is completed and multiple word clusters are obtained.

[0193] In an embodiment of the present application, a first word vector matrix corresponding to words in a candidate vocabulary set that is different from general words in the original text corpus in the vertical field is first screened out, and then the word vectors in the word vector matrix are clustered to obtain word clusters of multiple categories, and then the word clusters belonging to the target field are screened from the multiple word clusters, and the word clusters belonging to the target field are organized into a target vocabulary of the target field, thereby providing vocabulary resources for the target field. In this way, candidate new words that do not exist in the general vocabulary are first screened out from the original text corpus, so that words that may be new words in the target field are preliminarily screened out, reducing the amount of calculation during clustering, and determining the first word vector matrix corresponding to the words in the candidate vocabulary set. The segmented words in the candidate vocabulary set are clustered based on the first word vector matrix, and then new words belonging to the target field are screened from the clustering results to form a target vocabulary. In this way, words with similar word vectors are clustered into a group through clustering, thereby dividing the words in the candidate vocabulary set into multiple component word clusters, so that the segmented word clusters belonging to the target field can be screened out from the multiple component word clusters. In this way, words that do not belong to the target field are deleted through clustering, and the words in the target field are further screened, thereby improving the accuracy of screening new words in the target field.

[0194] Fig.11 FIG. 1 is a schematic diagram of an electronic device provided by an exemplary embodiment of the present application. Fig.11 As shown, the electronic device 11 of this embodiment includes: a processor 110, a memory 111, and a computer program 112 stored in the memory 111 and executable on the processor 110, such as a word discovery program. When the processor 110 executes the computer program 112, the steps in the above-mentioned various word discovery method embodiments are implemented, such as Figure 1 Alternatively, when the processor 110 executes the computer program 112, the functions of each unit in the above-mentioned device embodiments are realized, for example Fig.10 The functions of units 1001 to 1004 are shown.

[0195] Exemplarily, the computer program 112 may be divided into one or more units, which are stored in the memory 111 and executed by the processor 110 to complete the present application. The one or more units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 112 in the electronic device 11. For example, the computer program 112 may be divided into an acquisition unit, a first determination unit, a clustering unit, and a formation unit, and the specific functions of each module are as follows:

[0196] An acquisition unit 1001 is used to acquire original text corpus in a target domain, where the target domain is a vertical domain of new words to be identified;

[0197] A first determining unit 1002 is used to determine a first word vector matrix based on the original text corpus, where the first word vector matrix is ​​a vector matrix of words in the candidate vocabulary set that are different from common words in the original text corpus;

[0198] A clustering unit 1003 is used to cluster the words in the candidate vocabulary set based on the first word vector matrix to obtain a plurality of word clusters;

[0199] The forming unit 1004 is used to form a target word library of the target field with the word clusters belonging to the target field among the multiple word clusters. The target word library is used to provide vocabulary resources for the target field.

[0200] The electronic device 11 may be any electronic device with a control function. The electronic device 11 may include, but is not limited to, a processor 110 and a memory 111. Those skilled in the art will appreciate that Fig.11 It is only an example of the electronic device 11 and does not constitute a limitation of the electronic device 11. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 11 may also include input and output devices, network access devices, buses, etc.

[0201] The processor 110 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0202] The memory 111 may be an internal storage unit of the electronic device 11, such as a hard disk or memory of the electronic device 11. The memory 111 may also be an external storage device of the electronic device 11, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 11. Further, the memory 111 may also include both an internal storage unit of the electronic device 11 and an external storage device. The memory 111 is used to store the computer program and other programs and data required by the terminal device. The memory 111 may also be used to temporarily store data that has been output or is to be output.

[0203] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0204] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0205] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0206] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0207] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0209] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0210] The embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0211] The embodiment of the present application also provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0212] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A word discovery method, characterized in that: The method comprises: Acquire original text corpus in a target domain, where the target domain is a vertical domain of new words to be identified; Determine a first word vector matrix based on the original text corpus, where the first word vector matrix is ​​a vector matrix of words in the candidate vocabulary set that are different from common words in the original text corpus; Clustering the words in the candidate vocabulary set based on the first word vector matrix to obtain multiple word clusters; The word clusters belonging to the target domain among the plurality of word clusters are combined into a target word library of the target domain, and the target word library is used to provide vocabulary resources for the target domain.

2. The method according to claim 1, characterized in that The determining a first word vector matrix based on the original text corpus includes: Performing word segmentation and parsing on the original text corpus to obtain an initial vocabulary set corresponding to the original text corpus; Acquire a preset vocabulary set, wherein the preset vocabulary set is a general vocabulary set; Deleting the word segments in the preset vocabulary set from the initial vocabulary set to obtain a candidate vocabulary set corresponding to the original text corpus; The word vectors of each word in the candidate vocabulary set are combined into the first word vector matrix.

3. The method according to claim 2, characterized in that Before forming the word vector of each word in the candidate vocabulary into the first word vector matrix, the method further includes: Determining a first word vector for each word segment in the initial vocabulary set; and determining a second word vector for each word segment in the preset vocabulary set; The step of forming the first word vector matrix from the word vectors of each word in the candidate vocabulary includes: Deleting the word vector that is the same as the second word vector from the plurality of first word vectors, to obtain a first word vector corresponding to each word in the candidate vocabulary set; The first word vector matrix is ​​formed by combining the first word vector corresponding to each word segment in the candidate vocabulary set.

4. The method according to claim 3, characterized in that The forming of the first word vector matrix from the first word vector corresponding to each word segment in the candidate vocabulary set comprises: Counting the semantic weight of each word in the initial vocabulary set, wherein the semantic weight indicates the importance of the word in the original text corpus; amplifying the feature difference between the first word vectors corresponding to each word in the candidate vocabulary set based on the semantic weight of each word in the initial vocabulary set; The enlarged multiple first word vectors are combined into the first word feature matrix.

5. The method according to claim 1, characterized in that The clustering of the words in the candidate vocabulary set based on the first word vector matrix to obtain multiple word clusters includes: Performing dimensionality reduction processing on the first word vector matrix to obtain a second word vector matrix; The word segments in the candidate vocabulary set are clustered based on the second word vector matrix to obtain multiple word segmentation clusters.

6. The method according to claim 5, characterized in that The step of performing dimensionality reduction processing on the first word vector matrix to obtain a second word vector matrix includes: Determine multiple singular values ​​of the first word vector matrix and left and right singular vectors corresponding to each singular value; Sorting the plurality of singular values ​​from large to small to obtain a singular value sequence consisting of the plurality of singular values; Determine a first target number of target singular values ​​from the singular value sequence; The inverse vectors of the left and right singular vectors corresponding to the target number of target singular values ​​are used to form the second word vector matrix.

7. The method according to claim 6, characterized in that The method further comprises: Determine the difference between the first word vector matrix and the second word vector matrix to obtain a third word vector matrix; Amplify the difference between each vector element in the third word vector matrix.

8. The method according to claim 1, characterized in that The clustering of the words in the candidate vocabulary set based on the first word vector matrix to obtain multiple word clusters includes: Determine similarity threshold; Clustering the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result; Amplifying the similarity threshold based on a preset incremental strategy, and based on the amplified similarity threshold, continuing to perform the step of clustering the candidate words in the first word vector matrix based on the similarity threshold to obtain a clustering result, and updating the clustering result; Until the clustering is completed, multiple word clusters are obtained.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the word discovery method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the word discovery method according to any one of claims 1 to 8 is implemented.

Images