Word segmentation method and device based on corpus

Abstract

The present invention relates to a word segmentation method based on corpus, comprising the following steps: obtaining a target corpus; splitting the target corpus into multiple sub-corpus segments according to punctuation marks; performing adjacent character truncation on each sub-corpus segment according to a preset rule to obtain multiple character combinations of different lengths; counting the number of times that multiple words of different lengths appear in user comments in a preset time period; and performing word segmentation on the target corpus according to the number of times that they appear. The present invention can perform word segmentation intelligently, has a faster new word discovery speed, and does not generate word segmentation ambiguity.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a corpus-based word segmentation method and device. Background Art

[0002] Word segmentation is the basis of current text analysis, and almost all text analysis-based applications require word segmentation.

[0003] The current word segmentation methods are all based on thesaurus, which requires manual filtering and screening of qualified keywords on a regular basis. Generally, they are based on thesaurus methods such as mmseg, ik, and paoding. The word segmentation method based on thesaurus has the following shortcomings: manual maintenance of thesaurus is required, the speed of new word discovery is slow, and word segmentation ambiguity is easy to occur. Summary of the invention

[0004] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a word segmentation method and device based on corpus.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Specifically, a corpus-based word segmentation method is proposed, including the following:

[0007] Obtain the target corpus;

[0008] Splitting the target corpus into multiple sub-corpus segments according to punctuation marks;

[0009] For each sub-corpus segment, adjacent characters are truncated according to a preset rule to obtain a plurality of character combinations of different lengths;

[0010] Counting the number of times a plurality of words of different lengths appear in user comments within a preset time period;

[0011] The target corpus is segmented according to the number of times each word appears.

[0012] Further, specifically, for each sub-corpus segment, adjacent characters are cut off according to a preset rule to obtain multiple character combinations of different lengths, following the following rules:

[0013] Each sub-corpus segment is assumed to be ABCA'B'C'..., and is divided into adjacent 2-character and adjacent 3-character combinations to obtain multiple 2-character and 3-character character combinations with AB, BC and ABC as basic structures, where A, B, C are arbitrary single characters.

[0014] Further, specifically, the user comments in the preset time period are the number of comments by all users within half a year.

[0015] Further, specifically, the target corpus is segmented according to the number of occurrences, including analyzing each basic structure to obtain the number n0 of occurrences of the word of ABC characters in a preset time period, the number n1 of occurrences of the combination of AB characters, and the number n2 of occurrences of the combination of BC characters;

[0016] If the difference between n1 and n2 is greater than the first threshold, AB is judged to be a word;

[0017] If the difference between n2 and n1 is greater than the first threshold, then A is judged to be a single word, BC is a word or BCA' is a word;

[0018] If the absolute value of the difference between n1 and n2 is not greater than the first threshold, it is determined that ABC is a word or ABCA' is a word.

[0019] Furthermore, the method further comprises:

[0020] When the difference between n2 and n1 is greater than the first threshold, the number of BCs and the number of BCA's are obtained. If the difference between the number of BCs and the number of BCA's is lower than the second threshold, BCA' is determined to be a word. If the difference between the number of BCs and the number of BCA's is not lower than the second threshold, BC is determined to be a word.

[0021] When the absolute value of the difference between n1 and n2 is not greater than the first threshold, the number of ABC and the number of ABCA' are obtained. If the difference between the number of ABC and the number of ABCA' is lower than the second threshold, ABCA' is judged to be a word. If the difference between the number of BC and the number of BCA' is not lower than the second threshold, ABC is judged to be a word.

[0022] Furthermore, the method also includes, while truncating adjacent characters of each sub-corpus segment according to preset rules to obtain multiple character combinations of different lengths, forming the character combinations into corresponding hash structures, and quickly counting the number of times the character combinations appear in user comments in a preset time period based on the formed hash structure.

[0023] The present invention further proposes a corpus-based word segmentation device, characterized in that it includes the following:

[0024] A target corpus acquisition module is used to acquire the target corpus;

[0025] A sub-segment splitting module is used to split the target corpus into multiple sub-segments according to punctuation marks;

[0026] The character combination acquisition module is used to perform segmentation of adjacent characters according to preset rules for each sub-corpus segment to obtain multiple character combinations of different lengths;

[0027] A quantity statistics module, used to count the number of times a plurality of words of different lengths appear in user comments in a preset time period;

[0028] The word segmentation module is used to segment the target corpus according to the number of times each word appears.

[0029] Furthermore, the device also includes:

[0030] The hash structure generation module is used to truncate adjacent characters of each sub-corpus segment according to preset rules to obtain multiple character combinations of different lengths, and also to form the character combinations into corresponding hash structures, and to quickly count the number of times the character combinations appear in user comments in a preset time period based on the formed hash structures.

[0031] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0032] The beneficial effects of the present invention are:

[0033] The present invention establishes a mathematical model of word segmentation for a corpus, performs segmentation of adjacent characters according to a preset rule for each sub-corpus segment to obtain a plurality of character combinations of different lengths; counts the number of times that the plurality of words of different lengths appear in user comments in a preset time period; and performs word segmentation on the target corpus according to the number of times that the words appear respectively. The present invention can perform word segmentation intelligently, has a faster speed of discovering new words, and does not generate word segmentation ambiguity. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other features of the present disclosure will become more apparent by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present disclosure represent the same or similar elements. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:

[0035] Figure 1 Shown is a flow chart of the word segmentation method based on corpus of the present invention; DETAILED DESCRIPTION

[0036] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0037] Reference Figure 1 , Embodiment 1, the present invention proposes a word segmentation method based on corpus, including the following:

[0038] Step 110: Obtain target corpus;

[0039] Step 120: split the target corpus into multiple sub-corpus segments according to punctuation marks;

[0040] Step 130: for each sub-corpus segment, adjacent characters are truncated according to a preset rule to obtain a plurality of character combinations of different lengths;

[0041] Step 140, counting the number of times that the words of different lengths appear in user comments in a preset time period;

[0042] Step 150: Segment the target corpus according to the number of times the words appear respectively.

[0043] As a preferred embodiment of the present invention, specifically, for each sub-corpus segment, adjacent characters are cut off according to a preset rule to obtain a plurality of character combinations of different lengths, following the following rules:

[0044] Each sub-corpus segment is assumed to be ABCA'B'C'..., and is divided into adjacent 2-character and adjacent 3-character combinations to obtain multiple 2-character and 3-character character combinations with AB, BC and ABC as basic structures, where A, B, C are arbitrary single characters.

[0045] As a preferred implementation manner of the present invention, specifically, the user comments in the preset time period are the number of comments by all users within half a year.

[0046] As a preferred embodiment of the present invention, specifically, segmenting the target corpus according to the number of occurrences respectively includes:

[0047] Analyze each basic structure to obtain the number of words with ABC characters appearing in a preset time period n0, the number of words with AB characters appearing n1, and the number of words with BC characters appearing n2;

[0048] If the difference between n1 and n2 is greater than the first threshold, AB is judged to be a word;

[0049] If the difference between n2 and n1 is greater than the first threshold, then A is judged to be a single word, BC is a word or BCA' is a word;

[0050] If the absolute value of the difference between n1 and n2 is not greater than the first threshold, it is determined that ABC is a word or ABCA' is a word.

[0051] Specifically, the first threshold is defined by the user. Generally speaking, if the threshold is more than 5 times or above, it can be considered that combination is more reasonable than separation;

[0052] In this preferred embodiment, the corpus for analysis is given first: "Everyone believes that reducing the burden is not just about reducing students' homework."

[0053] First, let the corpus be split by punctuation marks, such as "Everyone believes that" and "Reducing the burden is not just about reducing the amount of students' homework"

[0054] Establishment of the mathematical model

[0055] Represent each character symbol as three letters A, B, and C, then

[0056] The three characters A, B, and C can be cut into two parts: AB and BC.

[0057] As a statistic on the corpus, the quantities can be recorded in Table 1 below

[0058]

[0059] Table 1

[0060] Case 1:

[0061] The quantity n1 is much larger than n2, indicating that the possibility of AB being a word is extremely high

[0062] Case 2:

[0063] The quantity n1 is much smaller than n2, indicating that A is very likely to be a single character, and BC may or may not be a word, and may be part of a subsequent word

[0064] Case 3:

[0065] If n1 and n2 do not differ much, then ABC may also be a word or part of a longer word. At this time, it cannot be definitely considered that ABC is a word, and the subsequent word needs to be added for overall analysis.

[0066] Specifically,

[0067] Example analysis 1: See Table 2

[0068] Combination of characters quantity Everyone (ABC) 2855 Everyone (AB) 10530 Jiadu (BC) 4442

[0069] Table 2

[0070] Based on the statistics of the corpus, it can be found that the number of times "everyone" appears in the corpus is much more than the number of times "family all" appears. Therefore, it can be determined that "everyone" is more likely to be a word, and "all" becomes a single character. That is

[0071] The word segmentation of "大家都" should be "大家" and "都".

[0072] Example analysis two: Refer to Table 3.

[0073] Combination of characters quantity All think (ABC) 101 Both recognized (AB) 271 Think (BC) 3547

[0074] Table 3

[0075] By the same token, in the analysis of this corpus, the number of occurrences of "认为" is much greater than that of "都认", that is, "认为" is more like a word, while "都" is more like a single character. The analysis method is exactly the same as that of "Example analysis one", which is convenient for recursive processing using a program.

[0076] Example analysis result:

[0077] "大家都认为" should be word-segmented into "大家", "都", and "认为". The word segmentation result is correct.

[0078] Taking the above word-segmented corpus as another example

[0079] Example analysis three (4 characters): Refer to Table 4.

[0080] Combination of characters quantity Not just (ABC) 387 Not only (AB) 977 Only (BC) 1302

[0081] Table 4

[0082] At this time, the number of occurrences of "不仅" and "仅仅" is not much different, and the number of occurrences of the three words "不仅仅" is also not small. Then, the subsequent character needs to be introduced for the above analysis. According to the above corpus,

[0083] Re-combine them into: "不仅仅是", and then re-analyze. At this time, any two of these characters can be regarded as a whole, and then re-combined into the digital model structure of ABC, that is: Refer to Table 5.

[0084]

[0085] Table 5

[0086] Final analysis result

[0087] Judging from the above analysis,

[0088] The combination method of AB is more suitable than that of BC. That is, "不仅仅" should be more of a word. The word segmentation is correct.

[0089] As a preferred implementation manner of the present invention, the method further includes

[0090] When the difference between n2 and n1 is greater than the first threshold, the number of BCs and the number of BCA's are obtained. If the difference between the number of BCs and the number of BCA's is lower than the second threshold, BCA' is determined to be a word. If the difference between the number of BCs and the number of BCA's is not lower than the second threshold, BC is determined to be a word.

[0091] When the absolute value of the difference between n1 and n2 is not greater than the first threshold, the number of ABC and the number of ABCA' are obtained. If the difference between the number of ABC and the number of ABCA' is lower than the second threshold, ABCA' is judged to be a word. If the difference between the number of BC and the number of BCA' is not lower than the second threshold, ABC is judged to be a word.

[0092] As a preferred embodiment of the present invention, the method also includes, while truncating adjacent characters of each sub-corpus segment according to preset rules to obtain multiple character combinations of different lengths, forming the character combinations into corresponding hash structures, and quickly counting the character combinations based on the formed hash structures.

[0093] In this preferred embodiment,

[0094] If we only segment words based on the corpus by traversing and matching, it will definitely be a very slow process. That is, we can make the corpus into a hash structure first, which can speed up the segmentation and make the analysis speed comparable to that based on the vocabulary. We can make any combination of characters in the corpus. We can also analyze according to the above corpus

[0095] “Reducing the burden on students is not just about reducing their homework”

[0096] In this corpus, character combinations with lengths ranging from 2 to 4 are used. See Table 6.

[0097]

[0098]

[0099] Table 6

[0100] By generating such permutations and combinations, all the corpus will generate such a hash structure, and the number can be counted very quickly.

[0101] The present invention further proposes a corpus-based word segmentation device, characterized in that it includes the following:

[0102] A target corpus acquisition module is used to acquire the target corpus;

[0103] A sub-segment splitting module is used to split the target corpus into multiple sub-segments according to punctuation marks;

[0104] The character combination acquisition module is used to perform segmentation of adjacent characters according to preset rules for each sub-corpus segment to obtain multiple character combinations of different lengths;

[0105] A quantity statistics module, used to count the number of times a plurality of words of different lengths appear in user comments in a preset time period;

[0106] The word segmentation module is used to segment the target corpus according to the number of times each word appears.

[0107] As a preferred embodiment of the present invention, the device further comprises:

[0108] The hash structure generation module is used to truncate adjacent characters of each sub-corpus segment according to preset rules to obtain multiple character combinations of different lengths, and also to form the character combinations into corresponding hash structures, and to quickly count the number of times the character combinations appear in user comments in a preset time period based on the formed hash structures.

[0109] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0110] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

[0111] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0112] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0113] Although the description of the present invention has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

[0114] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. Corpus-based word segmentation method, characterized in that: Includes the following: Obtain the target corpus; Splitting the target corpus into multiple sub-corpus segments according to punctuation marks; For each sub-corpus segment, adjacent characters are truncated according to a preset rule to obtain a plurality of character combinations of different lengths; Counting the number of times that a plurality of character combinations of different lengths appear in user comments in a preset time period; Segmenting the target corpus according to the number of times each word appears; Specifically, for each sub-corpus segment, adjacent characters are cut off according to preset rules to obtain multiple character combinations of different lengths, following the following rules: Assume each sub-segment to be ABCA ‘ B ‘ C ‘ ..., dividing according to adjacent 2 characters and adjacent 3 characters, to obtain multiple 2-character and 3-character character combinations with AB, BC and ABC as basic structures, where A, B, C are any single characters; Specifically, the target corpus is segmented according to the number of occurrences, including: Analyze each basic structure to obtain the number of words with ABC characters appearing in a preset time period n0, the number of words with AB characters appearing n1, and the number of words with BC characters appearing n2; If the difference between n1 and n2 is greater than the first threshold, AB is judged to be a word; If the difference between n2 and n1 is greater than the first threshold, then A is judged to be a single word, BC is a word or BCA ‘ for words; If the absolute value of the difference between n1 and n2 is not greater than the first threshold, then ABC is judged to be a word or ABCA ‘ for words; The method further comprises, When the difference between n2 and n1 is greater than the first threshold, the number of BCs and BCA are obtained. ‘ If the number of BC is equal to BCA ‘ If the difference in the number of is lower than the second threshold, then BCA is judged ‘ For words, if the number of BC is equal to BCA ‘ If the difference in the number of is not less than the second threshold, BC is judged to be a word; When the absolute value of the difference between n1 and n2 is not greater than the first threshold, the number of ABC and ABCA are obtained. ‘ If the number of ABC is equal to ABCA ‘ The difference in the number of is lower than the second threshold, then it is judged that ABCA ‘ For words, if the number of BC is equal to BCA ‘ If the difference in the number of is not lower than the second threshold, ABC is judged to be a word.

2. The word segmentation method based on corpus according to claim 1, characterized in that: Specifically, the user comments in the preset time period are the number of comments by all users within half a year.

3. The word segmentation method based on corpus according to claim 1, characterized in that: The method also includes, while truncating adjacent characters according to preset rules for each sub-corpus segment to obtain multiple character combinations of different lengths, forming the character combinations into corresponding hash structures, and quickly counting the number of times the character combinations appear in user comments in a preset time period based on the formed hash structures.

4. A word segmentation device based on corpus, characterized in that: Includes the following: A target corpus acquisition module is used to acquire the target corpus; A sub-segment splitting module is used to split the target corpus into multiple sub-segments according to punctuation marks; The character combination acquisition module is used to perform segmentation of adjacent characters according to preset rules for each sub-corpus segment to obtain multiple character combinations of different lengths; A quantity statistics module, used to count the number of times a plurality of character combinations of different lengths appear in user comments in a preset time period; A word segmentation module, used for segmenting the target corpus according to the number of times each word appears; Specifically, for each sub-corpus segment, adjacent characters are cut off according to preset rules to obtain multiple character combinations of different lengths, following the following rules: Assume each sub-segment to be ABCA ‘ B ‘ C ‘ ..., dividing according to adjacent 2 characters and adjacent 3 characters, to obtain multiple 2-character and 3-character character combinations with AB, BC and ABC as basic structures, where A, B, C are any single characters; Specifically, the target corpus is segmented according to the number of occurrences, including: Analyze each basic structure to obtain the number of words with ABC characters appearing in a preset time period n0, the number of words with AB characters appearing n1, and the number of words with BC characters appearing n2; If the difference between n1 and n2 is greater than the first threshold, AB is judged to be a word; If the difference between n2 and n1 is greater than the first threshold, then A is judged to be a single word, BC is a word or BCA ‘ for words; If the absolute value of the difference between n1 and n2 is not greater than the first threshold, then ABC is judged to be a word or ABCA ‘ for words; The method further comprises, When the difference between n2 and n1 is greater than the first threshold, the number of BCs and BCA are obtained. ‘ If the number of BC is equal to BCA ‘ If the difference in the number of is lower than the second threshold, then BCA is judged ‘ For words, if the number of BC is equal to BCA ‘ If the difference in the number of is not less than the second threshold, BC is judged to be a word; When the absolute value of the difference between n1 and n2 is not greater than the first threshold, the number of ABC and ABCA are obtained. ‘ If the number of ABC is equal to ABCA ‘ The difference in the number of is lower than the second threshold, then it is judged that ABCA ‘ For words, if the number of BC is equal to BCA ‘ If the difference in the number of is not lower than the second threshold, ABC is judged to be a word.

5. The word segmentation device based on corpus according to claim 4 is characterized in that: The device further comprises: The hash structure generation module is used to truncate adjacent characters of each sub-corpus segment according to preset rules to obtain multiple character combinations of different lengths, and also to form the character combinations into corresponding hash structures, and to quickly count the number of times the character combinations appear in user comments in a preset time period based on the formed hash structures.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images