Domain dictionary construction method and device
By segmenting the domain corpus and calculating the correlation index value, screening and combining the dictionary to construct the dictionary, the problems of low efficiency and limitation of the use conditions in the existing technology are solved, and efficient and accurate domain dictionary construction is achieved.
Patent Information
- Application Number
- CN202310774548.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-28
- Publication Date
- 2024-12-31
AI Technical Summary
The existing domain dictionary construction method is inefficient and has limitations on usage conditions, and requires domain word accumulation and manual annotation.
By segmenting the domain corpus, calculating the correlation index value of character fragments, and constructing dictionaries based on threshold filtering and combining, realizing domain dictionary construction without word accumulation and manual annotation.
It improves the efficiency and accuracy of domain dictionary construction, reduces omissions in the dictionary construction process, and realizes low-cost and efficient domain dictionary mining and construction.
Smart Images
Figure CN119227675A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of language processing technology, and in particular to a method and device for constructing a domain dictionary. Background Art
[0002] Words are important language units in the language system. Words are the basis for text processing and have richer expressive power than characters in semantic representation. Words have domain characteristics. Different fields have different vocabulary systems. For example, there are great differences between the military field, the medical field, the public security field, and the financial field. This domain characteristic can be used for domain classification, domain labeling, domain differentiation, keyword extraction, and feature word extraction. It can also support the construction of knowledge graph knowledge systems such as ontology and instance systems. On the other hand, it also proposes a number of tasks such as the extraction of domain words, the extraction of semantic associations between domain words, the expansion of domain words, and the construction of a domain word knowledge base.
[0003] Different domain dictionaries or domain lexicons can support different applications. For example, domain dictionaries can be subdivided into six types: domain feature dictionaries, domain stop dictionaries, domain sentiment dictionaries, domain abstract dictionaries, domain semantic dictionaries, and domain event lexicons. Currently, a commonly used method for constructing domain dictionaries is a statistical domain dictionary construction method, for example, Figure 1 As shown in the figure, the first step is to annotate the domain text with the accumulated domain words; the second step is to use the entity recognition task to train the LSTM, BERT and other models; the third step is to use the trained model to identify the domain words in the data to be identified; the fourth step is to filter the domain words identified in the third step by means of confidence, word frequency and other methods; the fifth step is to add the filtered domain words to the domain dictionary. It can be seen that the above-mentioned statistical-based domain dictionary construction method not only requires a certain amount of domain word accumulation and manual annotation, but also requires a high recognition accuracy of the model. In other words, the existing domain dictionary construction method is not only inefficient, but also has restrictions on usage conditions. Summary of the invention
[0004] The embodiment of the present invention provides a method and device for constructing a domain dictionary, so as to solve the problems in the prior art of low efficiency in constructing a domain dictionary and the existence of usage condition restrictions.
[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:
[0006] In a first aspect, an embodiment of the present invention provides a method for constructing a domain dictionary. The method comprises:
[0007] Segmenting at least one domain corpus to obtain a first character segment set, wherein the first character segment set includes at least one first character segment;
[0008] Calculating a correlation index value between characters of the first character segment;
[0009] Determine a second character segment set according to the relevance index value, the second character segment set including first character segments whose relevance index value is greater than or equal to a preset threshold;
[0010] determining a third character segment set according to the second character segment set, the third character segment set including the second character segment set and a second character segment of the at least one domain corpus, the second character segment being a character segment in the at least one domain corpus that is the same as a character segment obtained by combining at least two character segments of the second character segment set;
[0011] A domain dictionary is constructed according to the third character segment set.
[0012] In a second aspect, an embodiment of the present invention further provides a domain dictionary construction device. The domain dictionary construction device includes:
[0013] A segmentation module, configured to segment at least one domain corpus to obtain a first character segment set, wherein the first character segment set includes at least one first character segment;
[0014] A first calculation module, used for calculating a correlation index value between characters of the first character segment;
[0015] A first determining module, configured to determine a second character segment set according to the relevance index value, wherein the second character segment set includes a first character segment whose relevance index value is greater than or equal to a preset threshold;
[0016] a second determining module, configured to determine a third character segment set according to the second character segment set, the third character segment set including the second character segment set and a second character segment of the at least one domain corpus, the second character segment being a character segment in the at least one domain corpus that is identical to a character segment obtained by combining at least two character segments of the second character segment set;
[0017] A construction module is used to construct a domain dictionary according to the third character segment set.
[0018] In a third aspect, an embodiment of the present invention further provides a domain dictionary construction device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned domain dictionary construction method when executed by the processor.
[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned domain dictionary construction method are implemented.
[0020] In an embodiment of the present invention, at least one domain corpus is segmented to obtain a first character segment set, the first character segment set including at least one first character segment; a correlation index value between characters of the first character segment is calculated; a second character segment set is determined according to the correlation index value, the second character segment set including first character segments whose correlation index values are greater than or equal to a preset threshold; a third character segment set is determined according to the second character segment set, the third character segment set including the second character segment set and a second character segment of the at least one domain corpus, the second character segment being a character segment in the at least one domain corpus that is the same as a character segment obtained by combining at least two character segments of the second character segment set; A domain dictionary is constructed based on the third character fragment set. As can be seen from the above, the embodiment of the present application can realize the mining and construction of the domain dictionary without the accumulation of domain words and manual annotation of domain words. Not only is the implementation relatively simple, but also the efficiency of constructing the domain dictionary can be improved. In addition, by performing screening processes such as calculating the correlation index value between characters of each first character fragment and comparing it with a preset threshold, the accuracy of the mined domain words can be guaranteed, and by also using the character fragments in at least one domain corpus that are the same as the character fragment obtained by combining at least two character fragments of the second character fragment set as the mined domain words, the domain words mined from the above-mentioned at least one domain corpus can be expanded, and the omission of domain words in at least one domain corpus can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0022] Figure 1 It is a schematic diagram of constructing a domain dictionary provided by the related technology;
[0023] Figure 2 is a flowchart of a method for constructing a domain dictionary provided by an embodiment of the present invention;
[0024] Figure 3 is a structural diagram of a domain dictionary construction device provided by an embodiment of the present invention;
[0025] Figure 4 It is a structural diagram of a domain dictionary construction device provided by another embodiment of the present invention. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] The embodiment of the present invention provides a method for constructing a domain dictionary. Figure 2 , Figure 2 1 is a flowchart of a method for constructing a domain dictionary provided by an embodiment of the present invention. The method can be applied to electronic devices, such as terminals (such as mobile terminals, computers, etc.) or servers. Figure 2 As shown, the domain dictionary construction method provided by the embodiment of the present invention includes the following steps:
[0028] Step 201: segment at least one domain corpus to obtain a first character segment set, wherein the first character segment set includes at least one first character segment.
[0029] In this embodiment, the above-mentioned domain corpus can be understood as corpus or text with domain characteristics. For example, for the construction of a dictionary in the medical field, the above-mentioned domain corpus can be corpus in the medical field; for the construction of a dictionary in the financial field, the above-mentioned domain corpus can be corpus in the financial field. In practical applications, a large amount of corpus in a specific field can be collected for dictionary construction in the field. In other words, the number of the above-mentioned domain corpus is usually large, for example, more than 500 or more than 1,000.
[0030] The above segmentation of at least one domain corpus obtains a first character segment set, wherein the segmentation length of the above domain corpus can be reasonably set according to the needs. For example, the above segmentation length can include 1, 2, 3 and 4, and accordingly, 1-character segments, 2-character segments, 3-character segments and 4-character segments can be obtained respectively. In other words, the above first character segment set can include 1-character segments, 2-character segments, 3-character segments and 4-character segments; or, the above segmentation length can include 2 and 3, and accordingly, 2-character segments and 3-character segments can be obtained respectively. In other words, the above first character segment set can include 2-character segments and 3-character segments. It can be understood that the above first character segment set includes character segments obtained by segmenting each domain corpus of the at least one domain corpus.
[0031] Exemplarily, an n-gram model can be used to segment each domain corpus of at least one domain corpus to obtain a first character segment set, and the first character segment set can include character segments obtained by segmenting each domain corpus of the at least one domain corpus. The value of n can be reasonably set according to actual needs. For example, the value of n can be 1, 2, 3 and 4, that is, 1-gram, 2-gram, 3-gram and 4-gram are used to segment the domain corpus to obtain 1-character segment, 2-character segment, 3-character segment and 4-character segment; or, the value of n can be 1, 2, 3, 4 and 5, that is, 1-gram, 2-gram, 3-gram, 4-gram and 5-gram are used to segment the domain corpus to obtain 1-character segment, 2-character segment, 3-character segment, 4-character segment and 5-character segment.
[0032] It should be noted that the above-mentioned 1-character fragment can be understood as a fragment including 1 character, the above-mentioned 2-character fragment can be understood as a fragment including 2 characters, the above-mentioned 3-character fragment can be understood as a fragment including 3 characters, the above-mentioned 4-character fragment can be understood as a fragment including 4 characters, the above-mentioned 5-character fragment can be understood as a fragment including 5 characters, and so on.
[0033] Step 202: Calculate the correlation index value between characters in the first character segment.
[0034] In this embodiment, the above-mentioned correlation index value is used to indicate the correlation between characters in the first character segment.
[0035] It can be understood that, when there are multiple first character segments in the first character segment set, the above step 202 can be understood as respectively calculating the correlation index value between the characters of each first character segment in the first character segment set.
[0036] In some optional embodiments, the above step 202 can be understood as calculating the correlation index value between the characters of each first character segment with a character length greater than 1 in the first character segment set.
[0037] Exemplarily, the occurrence probability or number of occurrences of each first character segment in the first character segment set and the occurrence probability or number of occurrences of each sub-segment obtained by dividing the first character segment can be counted respectively, and then the correlation index value between characters of the first character segment can be calculated according to the occurrence probability or number of occurrences of each first character segment and the occurrence probability or number of occurrences of each sub-segment obtained by dividing the first character segment. Among them, the above-mentioned occurrence probability can be understood as the ratio of the number of occurrences to the total number of character segments in the first character segment set.
[0038] It can be understood that the stronger the correlation between the characters in the first character segment, the greater the probability that the first character segment is a word.
[0039] Step 203: Determine a second character segment set according to the relevance index value, wherein the second character segment set includes first character segments whose relevance index value is greater than or equal to a preset threshold.
[0040] It can be understood that the above-mentioned step 203 can be understood as determining the second character fragment set based on the correlation index value between characters of each first character fragment in the first character fragment set, and the second character fragment set includes the first character fragment in the first character fragment set whose correlation index value is greater than or equal to the corresponding preset threshold.
[0041] The above-mentioned preset threshold can be reasonably set according to actual needs. In some optional embodiments, all the first character segments of the above-mentioned first character segment set can correspond to the same preset threshold. In some optional embodiments, the character segments of the same length in the above-mentioned first character segment set can correspond to the same preset threshold, and the character segments of different lengths in the above-mentioned first character segment set can correspond to different preset thresholds.
[0042] Exemplarily, the relevance index value of each first character fragment in the first character fragment set can be compared with its corresponding preset threshold value. If the relevance index value of a first character fragment is greater than or equal to its corresponding preset threshold value, the first character fragment can be added to the second character fragment set, wherein the initial second character fragment set is an empty set.
[0043] Step 204: determine a third character fragment set based on the second character fragment set, the third character fragment set including the second character fragment set and a second character fragment of the at least one domain corpus, the second character fragment being a character fragment in the at least one domain corpus that is identical to a character fragment obtained by combining at least two character fragments of the second character fragment set.
[0044] In one embodiment, if the maximum segmentation length used in the above step 201 is the first segmentation length, then multiple second segmentation lengths greater than the first segmentation length can be used to slice each domain corpus to obtain character segments of different lengths. Optionally, the minimum second segmentation length of the above multiple second segmentation lengths can be the above first segmentation length plus 1, and the maximum segmentation length of the above multiple second segmentation lengths can be reasonably set according to actual needs, for example, it can be the length of the domain corpus, or it can be a preset value (for example, 5, 6 or 7, etc.). Then, the character segments obtained by segmenting the multiple second segmentation lengths can be matched with the character segments obtained by combining any multiple first character segments in the second character segment set, and the successfully matched character segments in the character segments obtained by segmenting the multiple second segmentation lengths (that is, the same character segments as the character segments obtained by combining the multiple first character segments) are added to the third character segment set.
[0045] In another embodiment, any multiple first character fragments of the above-mentioned second character fragment set can be combined respectively, and each character fragment obtained by the combination can be matched with the above-mentioned each domain corpus respectively. If a domain corpus contains a character fragment that is the same as a character fragment obtained by the combination, then the character fragment can be added to the third character fragment set.
[0046] For example, the above-mentioned second character segment set includes: {World Cup, football, football team, World Cup, FIFA}, and a certain field corpus is: Football World Cup welcomes you to join. Then, for the character segment "Football World Cup" of the field corpus, it is the same as the character segment obtained by combining "football" and "World Cup" in the above-mentioned second character segment set. Then, the second character segment "Football World Cup" can be added to the third character segment set.
[0047] It should be noted that, when combining multiple first character fragments, deduplication processing is required for the same parts within the multiple first character fragments. For example, the above-mentioned second character fragment set includes: {brain therapy device, therapy device, leukocytosis, cytopathic disease, non-genetically modified, genetically modified}, then the character fragment obtained by combining "brain therapy device" and "therapy device" is "brain therapy device".
[0048] Step 205: construct a domain dictionary according to the third character segment set.
[0049] Exemplarily, the character segments in the third character segment set can be directly used to construct a domain dictionary, that is, the character segments in the third character segment set can be used as domain words in the domain dictionary; or, the character segments in the third character segment set can be filtered, and a domain dictionary can be constructed based on the character segments in the filtered third character segment set, that is, the character segments in the filtered third character segment set can be used as domain words in the domain dictionary.
[0050] The domain dictionary construction method of the embodiment of the present invention divides at least one domain corpus to obtain a first character segment set, wherein the first character segment set includes at least one first character segment; calculates a correlation index value between characters of the first character segment; determines a second character segment set according to the correlation index value, wherein the second character segment set includes first character segments whose correlation index values are greater than or equal to a preset threshold; determines a third character segment set according to the second character segment set, wherein the third character segment set includes the second character segment set and a second character segment of the at least one domain corpus, wherein the second character segment is a character in the at least one domain corpus that is the same as a character segment obtained by combining at least two character segments of the second character segment set. character fragments; constructing a domain dictionary based on the third character fragment set. It can be seen from the above that the embodiment of the present application can realize the mining and construction of the domain dictionary without the accumulation of domain words and the manual annotation of domain words, which is not only relatively simple to implement but also can improve the efficiency of constructing the domain dictionary; in addition, by performing screening processes such as calculating the correlation index value between characters and comparing with a preset threshold value on each first character fragment, the accuracy of the mined domain words can be guaranteed, and by taking the character fragments in at least one domain corpus that are the same as the character fragment obtained by combining at least two character fragments of the second character fragment set as the mined domain words, the domain words mined from the above-mentioned at least one domain corpus can be expanded, and the omission of domain words in at least one domain corpus can be reduced.
[0051] Optionally, calculating the correlation index value between characters of the first character segment includes:
[0052] calculating a sub-association index value for each sub-segment combination of the first character segment, wherein the sub-segment combination includes at least two sub-segments in the first character segment;
[0053] The correlation index value between characters of the first character segment is calculated according to the sub-correlation index value.
[0054] It can be understood that the correlation index value of each first character segment of the first character segment set can be calculated using the correlation index value calculation method provided in this embodiment. In addition, each sub-segment combination of the above-mentioned first character segment includes at least two sub-segments of the first character segment, and different sub-segment combinations include different sub-segments.
[0055] Optionally, at least two sub-segments included in each sub-segment combination of the first character segment can be combined to obtain the first character segment. For example, for the first character segment "abc", its corresponding sub-segment combination can include {a, bc} and {ab, c}; for the first character segment "abcd", its corresponding sub-segment combination can include {a, bcd}, {ab, cd}, {abc, d}.
[0056] Exemplarily, the occurrence probabilities of each first character segment in the first character segment set and the occurrence probabilities of each sub-segment in each sub-segment combination of each first character segment can be calculated respectively, and then the sub-association index values of the first character segment in each sub-segment combination can be calculated based on the occurrence probabilities of each first character segment and the occurrence probabilities of each sub-segment in each sub-segment combination of the first character segment. For example, the sub-association index values of the first character segment in each sub-segment combination can be calculated based on the following calculation formula:
[0057]
[0058] Wherein, p(i) represents the probability of occurrence of the i-th first character segment in the first character segment set, J k represents the number of sub-segments of the kth seed segment combination of the i-th first character segment, p(j) represents the number of occurrences of the jth sub-segment of the kth seed segment combination in the first character segment set, P ik Represents the sub-correlation index value of the kth seed segment combination of the i-th first character segment, where i and k are both positive integers.
[0059] After obtaining the sub-correlation index values of the sub-segment combinations of the first character segment, the correlation index value between the characters of the first character segment can be determined based on the multiple sub-correlation index values of the first character segment. For example, the average value of the multiple sub-correlation index values of the first character segment can be determined as the correlation index value of the first character segment, or the middle value of the multiple sub-correlation index values of the first character segment can be determined as the correlation index value of the first character segment, or the minimum value of the multiple sub-correlation index values of the first character segment can be determined as the correlation index value of the first character segment, and so on.
[0060] For example, for the first character fragment "abc", its corresponding sub-segment combinations include {a, bc} and {ab, c}. The sub-association index value of the sub-segment combination {a, bc} of the first character fragment "abc" and the sub-association index value of the sub-segment combination {ab, c} can be calculated respectively, and the minimum value of the sub-association index value under the sub-segment combination {a, bc} and the sub-association index value under the sub-segment combination {ab, c} can be used as the association index value of the first character fragment "abc".
[0061] The embodiment of the present invention calculates the sub-correlation index values of various sub-segment combinations of each first character segment respectively, and calculates the correlation index value of each first character segment according to the sub-correlation index value of each sub-segment combination of each first character segment respectively, that is, the correlation index value of the first character segment is determined by comprehensively considering the sub-correlation index values of multiple sub-segment combinations of the first character segment, which is beneficial to ensure that the determined correlation index value of the first character segment can more objectively and accurately reflect the correlation between characters in the first character segment.
[0062] Optionally, calculating the sub-association index value of each sub-segment combination of the first character segment includes:
[0063] The sub-association index value of each sub-segment combination of the first character segment is calculated based on a first calculation formula, and the first calculation formula is as follows:
[0064]
[0065] Wherein, NUM(i) represents the number of occurrences or the probability of occurrence of the i-th first character segment in the first character segment set, J k represents the number of sub-segments of the kth seed segment combination of the i-th first character segment, NUM(j) represents the number of occurrences or the probability of occurrence of the jth sub-segment of the kth seed segment combination in the first character segment set, P ik Represents the sub-correlation index value of the kth seed segment combination of the i-th first character segment, where i and k are both positive integers.
[0066] For example, the i-th first character segment of the first character segment set is "abc", NUM(i) represents the number of occurrences or the probability of occurrence of the first character segment "abc" in the first character segment set, and the above-mentioned first character segment "abc" includes two sub-segment combinations, namely {a, bc} and {ab, c}, then the values of the above-mentioned k are 1 and 2 respectively, J1 represents the number of sub-segments of the first seed segment combination {a, bc}, J2 represents the number of sub-segments of the second seed segment combination {ab, c}, for k=1, NUM(j) represents the number of occurrences or the probability of occurrence of the sub-segments {a} and {bc} in the first character segment set, and for k=2, NUM(j) represents the number of occurrences or the probability of occurrence of the sub-segments {ab} and {c} in the first character segment set.
[0067] The embodiment of the present invention calculates the sub-correlation index value of each sub-segment combination of each of the first character fragments based on the first calculation formula, that is, the sub-correlation index value is calculated by comprehensively considering the number of occurrences or the probability of occurrence of the first character fragment and the number of occurrences or the probability of occurrence of each sub-segment under the sub-segment combination of the first character fragment. This is not only relatively simple to implement, but also can ensure that the calculated correlation index value of the first character fragment can more accurately reflect the correlation between the characters of the first character fragment.
[0068] Optionally, calculating the correlation index value between characters of the first character segment according to the sub-correlation index value includes:
[0069] The minimum sub-association index value among the sub-association index values is determined as the association index value between characters of the first character segment.
[0070] In this embodiment, after calculating the sub-correlation index values of each sub-segment combination of the first character segment, the minimum sub-correlation index value among all the sub-correlation index values may be determined as the correlation index value between characters of the first character segment.
[0071] For example, the sub-association index value of the first seed segment combination {a, bc} of the first character segment "abc" is P1, and the sub-association index value of the second seed segment combination {ab, c} of the first character segment "abc" is P2, then the association index value of the first character segment "abc" can be determined as the smaller value between P1 and P2.
[0072] Exemplarily, the correlation index value between characters of the first character segment may be calculated based on a second calculation formula, and the second calculation formula is as follows:
[0073]
[0074] Wherein, NUM(i) represents the number of occurrences of the i-th first character segment of the first character segment set in the first character segment set, I represents the number of sub-segment combinations obtained by dividing the i-th first character segment, each of which includes at least two sub-segments obtained by dividing the i-th first character segment, and J k represents the kth seed segment combination of the i-th first character segment, NUM(j) represents the number of occurrences of the jth sub-segment of the k-th seed segment combination in the first character segment set, P ik It represents the correlation index value of the i-th first character segment under the k-th seed segment combination, and min represents the minimum value.
[0075] This embodiment determines the correlation index value between characters of each of the first character fragments as the correlation index value between characters of each of the first character fragments by respectively determining the minimum sub-correlation index value among the sub-correlation index values of each sub-fragment combination of each of the first character fragments. In this way, when the correlation index value is compared with the corresponding preset threshold value to perform first character fragment screening, if the minimum sub-correlation index value satisfies the corresponding preset threshold value, it can be ensured that the screened first character fragment has a higher probability of forming a word.
[0076] Optionally, segmenting at least one domain corpus to obtain a first character segment set includes:
[0077] Using different segmentation lengths to segment the at least one domain corpus to obtain a first character segment set;
[0078] Among them, the preset thresholds corresponding to first character segments of different lengths are different.
[0079] The inventors of the present invention have discovered that the distribution of correlation index values of character fragments of different lengths is often different. This embodiment sets different preset thresholds for character fragments of different lengths, which is conducive to more accurately screening out character fragments with a higher probability of forming words.
[0080] Optionally, the preset threshold corresponding to the first character segment is positively correlated with the length of the first character segment.
[0081] The preset threshold corresponding to the above-mentioned first character fragment is positively correlated with the length of the first character fragment. It can be understood that the longer the length of the above-mentioned first character fragment, the larger the preset threshold corresponding to the first character fragment, and the shorter the length of the above-mentioned first character fragment, the smaller the preset threshold corresponding to the first character fragment.
[0082] For example, the preset threshold corresponding to a 2-character segment is smaller than the preset threshold corresponding to a 3-character segment, the preset threshold corresponding to a 3-character segment is smaller than the preset threshold corresponding to a 4-character segment, and so on.
[0083] The inventors of the present invention have discovered that the longer the character fragment is, the greater the probability that the relevance index value is biased toward the larger value. Therefore, in this embodiment, for first character fragments with longer lengths, a larger corresponding preset threshold is set. This is conducive to more accurately screening out character fragments with a higher probability of forming words.
[0084] Optionally, the segmenting of at least one domain corpus by using different segmentation lengths to obtain a first character segment set includes:
[0085] Each of the multiple domain corpora is segmented using a 1-gram model, a 2-gram model, a 3-gram model, and a 4-gram model to obtain a first character segment set.
[0086] The inventors of the present invention have found that the length of a word is usually 1 to 4. Therefore, the embodiment of the present invention adopts 1-gram model, 2-gram model, 3-gram model and 4-gram model to segment each of multiple domain corpora to obtain a first character segment set. This can not only ensure that the first character segment set can cover the vast majority of domain words in the above-mentioned multiple domain corpora, but also reduce the number of interference words, which is conducive to improving the accuracy of the mined domain words. In addition, it can also reduce the processing amount and improve processing efficiency.
[0087] Optionally, the maximum segmentation length of the domain corpus is L, where L is a positive integer; and constructing a domain dictionary according to the third character segment set includes:
[0088] Eliminate a third character segment from the third character segment set, where the third character segment satisfies any of the following conditions: the length is less than or equal to L, and the third character segment does not belong to the second character segment set; the length is greater than L, and at least one of the at least two L character segments obtained by segmentation using the maximum segmentation length does not belong to the second character segment set;
[0089] A domain dictionary is constructed according to the set of third character segments after excluding the third character segment.
[0090] In this embodiment, the maximum segmentation length of the domain corpus can be reasonably set according to actual needs, for example, L is 3 or 4.
[0091] For each character segment of the third character segment set, if the character segment satisfies any of the following conditions, the character segment may be removed from the third character segment set:
[0092] The length is less than and equal to L, and does not belong to the second character segment set;
[0093] The length is greater than L, and at least one of the at least two L character segments obtained by segmentation using the maximum segmentation length does not belong to the second character segment set.
[0094] Taking L equal to 4 as an example, for example, the above-mentioned second character segment set includes: {World Cup, Football, Football Team, World Cup, FIFA}, for the character segment "Football World Cup" of the third character segment set, the 4-character segments obtained by 4-gram segmentation include "Football World" and "Football World Cup". Since "Football World" and "Football World Cup" do not belong to the above-mentioned second character segment set, the character segment "Football World Cup" needs to be removed from the third character segment set.
[0095] For another example, the second character segment set includes: {brain therapy device, therapy device, leukocytosis, cell pathology, non-genetically modified, genetically modified}. For the character segment “brain therapy device” of the third character segment set, the 4-character segments obtained through 4-gram segmentation include “brain therapy device” and “physiotherapy device”. Since the “brain therapy device” and “physiotherapy device” both belong to the second character segment set, the character segment “brain therapy device” is retained in the third character segment set; for the character segment “leukocytosis” of the third character segment set, the 4-character segments obtained through 4-gram segmentation include “leukocytosis” and “cell pathology”. Since the “leukocytosis” and “cell pathology” both belong to the second character segment set, the character segment “leukocytosis” is retained in the third character segment set.
[0096] The embodiment of the present invention removes the third character segment from the third character segment set and constructs a domain dictionary based on the third character segment set after removing the third character segment, which can further improve the accuracy of the mined domain words.
[0097] In summary, the domain dictionary construction method provided in the embodiment of the present invention realizes the construction of the domain dictionary through domain corpus segmentation (for example, domain corpus segmentation based on the n-gram model), relevance index value calculation, threshold judgment (i.e., obtaining the first character segment in the first character segment set whose relevance index value is greater than or equal to the corresponding preset threshold), character segment matching (i.e., determining the third character segment set based on the second character segment set), character segment screening (i.e., eliminating the third character segment from the third character segment set), etc., which has the advantages of being relatively simple and efficient, can save a lot of manpower and material resources, and realize the construction of the domain dictionary at low cost. In addition, the relevance index value calculation method provided in the implementation of the present invention combined with the threshold judgment and character segment screening parts can improve the quality or accuracy of the domain dictionary. In addition, the character segment matching part provided in the embodiment of the present invention can mine character segments whose length is greater than the maximum length of the character segments of the first character segment set, thereby reducing the omission of domain words.
[0098] See also Figure 3 , Figure 3 is a structural diagram of a domain dictionary construction device provided by an embodiment of the present invention. Figure 3 As shown, the domain dictionary construction device 300 includes:
[0099] A segmentation module 301 is used to segment at least one domain corpus to obtain a first character segment set, wherein the first character segment set includes at least one first character segment;
[0100] A first calculation module 302, used to calculate the correlation index value between characters of the first character segment;
[0101] A first determining module 303, configured to determine a second character segment set according to the relevance index value, wherein the second character segment set includes first character segments whose relevance index value is greater than or equal to a preset threshold;
[0102] A second determining module 304 is configured to determine a third character segment set according to the second character segment set, the third character segment set including the second character segment set and a second character segment of the at least one domain corpus, the second character segment being a character segment in the at least one domain corpus that is identical to a character segment obtained by combining at least two character segments of the second character segment set;
[0103] The construction module 305 is used to construct a domain dictionary according to the third character segment set.
[0104] Optionally, the first calculation module includes:
[0105] a first calculating unit, configured to calculate a sub-association index value of each sub-segment combination of the first character segment, wherein the sub-segment combination includes at least two sub-segments of the first character segment;
[0106] The second calculation unit is used to calculate the correlation index value between characters of the first character segment according to the sub-correlation index value.
[0107] Optionally, the first computing unit is specifically configured to:
[0108] The sub-association index value of each sub-segment combination of the first character segment is calculated based on a first calculation formula, and the first calculation formula is as follows:
[0109]
[0110] Wherein, NUM(i) represents the number of occurrences or the probability of occurrence of the i-th first character segment in the first character segment set, J k represents the number of sub-segments of the kth seed segment combination of the i-th first character segment, NUM(j) represents the number of occurrences or the probability of occurrence of the jth sub-segment of the kth seed segment combination in the first character segment set, P ik Represents the sub-correlation index value of the kth seed segment combination of the i-th first character segment, where i and k are both positive integers.
[0111] Optionally, the second computing unit is specifically configured to:
[0112] The minimum sub-association index value among the sub-association index values is determined as the association index value between characters of the first character segment.
[0113] Optionally, the segmentation module is specifically used for:
[0114] Using different segmentation lengths to segment the at least one domain corpus to obtain a first character segment set;
[0115] Among them, the preset thresholds corresponding to first character segments of different lengths are different.
[0116] Optionally, the preset threshold corresponding to the first character segment is positively correlated with the length of the first character segment.
[0117] Optionally, the maximum segmentation length of the domain corpus is L, where L is a positive integer;
[0118] The building blocks are specifically used for:
[0119] Eliminate a third character segment from the third character segment set, where the third character segment satisfies any of the following conditions: the length is less than or equal to L, and the third character segment does not belong to the second character segment set; the length is greater than L, and at least one of the at least two L character segments obtained by segmentation using the maximum segmentation length does not belong to the second character segment set;
[0120] A domain dictionary is constructed according to the set of third character segments after excluding the third character segment.
[0121] The domain dictionary construction device 300 provided in the embodiment of the present invention can implement each process in the above device embodiment, and will not be described again here to avoid repetition.
[0122] See also Figure 4 , Figure 4 is a structural diagram of a domain dictionary construction device provided by another embodiment of the present invention, such as Figure 4 As shown, the domain dictionary construction device 400 includes: a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor. Each component in the domain dictionary construction device 400 is coupled together via a bus interface 403. When the computer program is executed by the processor 401, the following steps are implemented:
[0123] Segmenting at least one domain corpus to obtain a first character segment set, wherein the first character segment set includes at least one first character segment;
[0124] Calculating a correlation index value between characters of the first character segment;
[0125] Determine a second character segment set according to the relevance index value, the second character segment set including first character segments whose relevance index value is greater than or equal to a preset threshold;
[0126] determining a third character segment set according to the second character segment set, the third character segment set including the second character segment set and a second character segment of the at least one domain corpus, the second character segment being a character segment in the at least one domain corpus that is the same as a character segment obtained by combining at least two character segments of the second character segment set;
[0127] A domain dictionary is constructed according to the third character segment set.
[0128] It should be understood that in the embodiment of the present invention, when the computer program is executed by the processor 501, each process in the above-mentioned embodiment of the domain dictionary construction device can be implemented, and in order to avoid repetition, it will not be described again here.
[0129] An embodiment of the present invention also provides a domain dictionary construction device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, each process of the above-mentioned domain dictionary construction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0130] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned domain dictionary construction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0131] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the method described in each embodiment of the present invention.
[0133] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.
Claims
1. A method for constructing a domain dictionary, characterized in that: include: Segmenting at least one domain corpus to obtain a first character segment set, wherein the first character segment set includes at least one first character segment; Calculating a correlation index value between characters of the first character segment; Determine a second character segment set according to the relevance index value, the second character segment set including first character segments whose relevance index value is greater than or equal to a preset threshold; determining a third character segment set according to the second character segment set, the third character segment set including the second character segment set and a second character segment of the at least one domain corpus, the second character segment being a character segment in the at least one domain corpus that is the same as a character segment obtained by combining at least two character segments of the second character segment set; A domain dictionary is constructed according to the third character segment set.
2. The method according to claim 1, characterized in that The calculating the correlation index value between characters of the first character segment includes: calculating a sub-association index value of each sub-segment combination of the first character segment, wherein the sub-segment combination includes at least two sub-segments of the first character segment; The correlation index value between characters of the first character segment is calculated according to the sub-correlation index value.
3. The method according to claim 2, characterized in that The calculating the sub-association index value of each sub-segment combination of the first character segment includes: The sub-association index value of each sub-segment combination of the first character segment is calculated based on a first calculation formula, and the first calculation formula is as follows: Wherein, NUM(i) represents the number of occurrences or the probability of occurrence of the i-th first character segment in the first character segment set, J k represents the number of sub-segments of the kth seed segment combination of the i-th first character segment, NUM(j) represents the number of occurrences or the probability of occurrence of the jth sub-segment of the kth seed segment combination in the first character segment set, P ik Represents the sub-correlation index value of the kth seed segment combination of the i-th first character segment, where i and k are both positive integers.
4. The method according to claim 2 or 3, characterized in that: The calculating the correlation index value between characters of the first character segment according to the sub-correlation index value includes: The minimum sub-association index value among the sub-association index values is determined as the association index value between characters of the first character segment.
5. The method according to claim 1 or 2, characterized in that: The step of segmenting at least one domain corpus to obtain a first character segment set includes: Using different segmentation lengths to segment the at least one domain corpus to obtain a first character segment set; Among them, the preset thresholds corresponding to first character segments of different lengths are different.
6. The method according to claim 5, characterized in that The preset threshold corresponding to the first character segment is positively correlated with the length of the first character segment.
7. The method according to claim 1, characterized in that The maximum segmentation length of the domain corpus is L, where L is a positive integer; and constructing a domain dictionary according to the third character segment set includes: Eliminate a third character segment from the third character segment set, where the third character segment satisfies any of the following conditions: the length is less than or equal to L, and the third character segment does not belong to the second character segment set; the length is greater than L, and at least one of the at least two L character segments obtained by segmentation using the maximum segmentation length does not belong to the second character segment set; A domain dictionary is constructed according to the set of third character segments after excluding the third character segment.
8. A domain dictionary construction device, characterized in that: include: A segmentation module, configured to segment at least one domain corpus to obtain a first character segment set, wherein the first character segment set includes at least one first character segment; A first calculation module, used for calculating a correlation index value between characters of the first character segment; A first determining module, configured to determine a second character segment set according to the relevance index value, wherein the second character segment set includes a first character segment whose relevance index value is greater than or equal to a preset threshold; a second determining module, configured to determine a third character segment set according to the second character segment set, the third character segment set including the second character segment set and a second character segment of the at least one domain corpus, the second character segment being a character segment in the at least one domain corpus that is identical to a character segment obtained by combining at least two character segments of the second character segment set; A construction module is used to construct a domain dictionary according to the third character segment set.
9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method for constructing a domain dictionary according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for constructing a domain dictionary according to any one of claims 1 to 7 are implemented.