Method, device and system for detecting reduplication errors
Calculate the probability of using text fragments by word segmentation, part-of-speech matching and language models, and solve the problem of low accuracy of word overlap error detection in the prior art, and realize high accuracy and high speed misuse judgment of word overlap, suitable for different lengths sentences.
Patent Information
- Application Number
- CN201911263865.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-11
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2039-12-11
AI Technical Summary
The prior art has low accuracy when detecting word repetition errors and is difficult to adapt to sentences of different lengths, resulting in low accuracy in judgment of word repetitions in long sentences.
By obtaining punctuation marks and string information in the text, word segmentation detects adjacent overlays and matches part of speech in the preset part of speech database. If the match is successful, deletes the overlays and uses the mobile window and language model to calculate the probability of using text fragments. Determine whether the word repetition is used incorrectly.
It improves the judgment accuracy and detection speed of misuse of word repetition, adapts to sentences of different lengths, and improves the accuracy of judgment of word repetition in long sentences.
Smart Images

Figure CN112949289B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer natural language processing, and in particular to a method, device and system for detecting reduplication errors. Background Art
[0002] With the rapid development of the Internet, people use a lot of text to obtain information in their daily lives. It is inevitable to use reduplication, especially to strengthen the tone of voice, such as fiery red maple leaves and let's be happy together. It is easy to make mistakes in the use of reduplication. Therefore, it is necessary to determine whether there are reduplication errors in the text.
[0003] Reduplication is a word formed by two or more characters with the same shape and meaning being used together. However, it is not easy to determine whether the repeated use of a word is a reduplication error, as not all words can be considered reduplication. For example, statistical recognition may treat reduplication as a typo, or use all words that meet the reduplication format as reduplication, which will result in many incorrect uses that go undetected.
[0004] Currently, there are few tools for detecting reduplication errors. Even if they can be detected, most of them rely on rules, resulting in low accuracy in identifying reduplication errors. Summary of the invention
[0005] The present invention provides a method, device and system for detecting reduplication errors, so as to realize the judgment of misuse of reduplication, improve the accuracy and speed of detection, and can adapt to sentences of different lengths, thereby improving the accuracy of reduplication judgment in long sentences.
[0006] In a first aspect, an embodiment of the present invention provides a method for detecting reduplication errors, comprising:
[0007] Acquire text, and obtain punctuation marks and character string information in the text;
[0008] Obtaining a plurality of texts to be detected according to the punctuation marks and the character string information, and obtaining repeated phrases in the texts to be detected;
[0009] Segmenting the original text segment containing the repeated phrase to detect whether there are adjacent reduplications;
[0010] If the adjacent reduplication exists, matching the part of speech of the adjacent reduplication in a preset part of speech database;
[0011] If the match is successful, a redundant word in the original text segment is deleted to obtain a text segment to be detected, and the usage probabilities of the original text segment and the text segment to be detected are obtained;
[0012] If the usage probability of the original text segment is less than the usage probability of the text segment to be detected, it is determined that the adjacent reduplicated words are used incorrectly in the original text segment.
[0013] In a possible design, obtaining the usage probabilities of the original text segment and the to-be-detected text segment includes:
[0014] Using a moving window to move on the original text segment and the to-be-detected text segment respectively, to obtain a plurality of target inspection characters without the reduplicated words;
[0015] The use probabilities of the target test word in the original text segment and the text segment to be detected are detected respectively by using a preset language model, and then the use probabilities of the original text segment and the text segment to be detected are determined.
[0016] In one possible design, it also includes:
[0017] Setting the parameters of the moving window according to the original text segment and the text segment to be detected;
[0018] According to the size parameter of the moving window, the moving window starts from the positions of the reduplications in the original text segment and the text segment to be detected, and moves to the positions of both ends in sequence to obtain a plurality of target inspection characters without the reduplications.
[0019] In one possible design, it also includes:
[0020] If the target inspection word does not fill the moving window, the moving window is determined to be at a partial position at one end of the original text segment and the text segment to be detected, and when the moving window is moved at a partial position at the other end, the target inspection word fills the moving window.
[0021] In a possible design, matching the parts of speech of the adjacent reduplicated words in a preset part of speech database includes:
[0022] Parts of speech are marked for the adjacent reduplications, and the parts of speech of the adjacent reduplications are matched with pre-stored parts of speech in a preset part-of-speech database.
[0023] In one possible design, it also includes:
[0024] If there is no pre-stored part of speech matching the adjacent reduplication word in the preset part of speech database, it is determined that the adjacent reduplication word is used incorrectly.
[0025] In a possible design, after determining that the adjacent reduplication words are used incorrectly in the original text segment, the method further includes:
[0026] Marks adjacent reduplications that are used incorrectly in the text.
[0027] In a second aspect, an embodiment of the present invention provides a device for detecting reduplication errors, comprising:
[0028] An acquisition module, used to acquire text and obtain punctuation marks and character string information in the text;
[0029] An obtaining module, used for obtaining a plurality of texts to be detected according to the punctuation marks and the string information, and obtaining repeated phrases in the texts to be detected;
[0030] A detection module, used to segment the original text segment where the repeated phrase is located, and detect whether there are adjacent reduplications;
[0031] A matching module, for matching the part of speech of the adjacent reduplication words in a preset part of speech database if the adjacent reduplication words exist;
[0032] A deletion module, for deleting a redundant word in the original text segment to obtain a text segment to be detected if the match is successful, and obtaining the usage probabilities of the original text segment and the text segment to be detected;
[0033] The determination module is used to determine that the adjacent reduplicated words are used incorrectly in the original text segment if the usage probability of the original text segment is less than the usage probability of the text segment to be detected.
[0034] In a possible design, obtaining the usage probabilities of the original text segment and the to-be-detected text segment includes:
[0035] Using a moving window to move on the original text segment and the to-be-detected text segment respectively, to obtain a plurality of target inspection characters without the reduplicated words;
[0036] The use probabilities of the target test word in the original text segment and the text segment to be detected are detected respectively by using a preset language model, and then the use probabilities of the original text segment and the text segment to be detected are determined.
[0037] In one possible design, it also includes:
[0038] Setting the parameters of the moving window according to the original text segment and the text segment to be detected;
[0039] According to the size parameter of the moving window, the moving window starts from the positions of the reduplications in the original text segment and the text segment to be detected, and moves to the positions of both ends in sequence to obtain a plurality of target inspection characters without the reduplications.
[0040] In one possible design, it also includes:
[0041] If the target inspection word does not fill the moving window, the moving window is determined to be at a partial position at one end of the original text segment and the text segment to be detected, and when the moving window is moved at a partial position at the other end, the target inspection word fills the moving window.
[0042] In a possible design, matching the parts of speech of the adjacent reduplicated words in a preset part of speech database includes:
[0043] Parts of speech are marked for the adjacent reduplications, and the parts of speech of the adjacent reduplications are matched with pre-stored parts of speech in a preset part-of-speech database.
[0044] In one possible design, it also includes:
[0045] If there is no pre-stored part of speech matching the adjacent reduplication word in the preset part of speech database, it is determined that the adjacent reduplication word is used incorrectly.
[0046] In a possible design, after determining that the adjacent reduplication words are used incorrectly in the original text segment, the method further includes:
[0047] Marks adjacent reduplications that are used incorrectly in the text.
[0048] In a third aspect, an embodiment of the present invention provides a system for detecting reduplication errors, comprising: a memory and a processor, wherein the memory stores executable instructions of the processor; wherein the processor is configured to execute the method for detecting reduplication errors described in any one of the first aspects by executing the executable instructions.
[0049] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is the method for detecting reduplication errors described in any one of the first aspects.
[0050] The present invention provides a method, device and system for detecting reduplication errors, the method comprising: obtaining a text, and obtaining punctuation marks and character string information in the text; obtaining a plurality of texts to be detected according to the punctuation marks and the character string information, and obtaining repeated phrases in the texts to be detected;
[0051] The original text segment where the repeated phrase is located is segmented to detect whether there are adjacent reduplications; if there are adjacent reduplications, the part of speech of the adjacent reduplications is matched in a preset part of speech database; if the match is successful, one of the reduplications in the original text segment is deleted to obtain a text segment to be detected, and the usage probability of the original text segment and the text segment to be detected is obtained; if the usage probability of the original text segment is less than the usage probability of the text segment to be detected, it is determined that the adjacent reduplications are used incorrectly in the original text segment. This can realize the judgment of the misuse of reduplications, improve the accuracy and speed of detection, and can adapt to sentences of different lengths, thereby improving the accuracy of reduplication judgment in long sentences. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0053] Figure 1 This is a schematic diagram of a typical application scenario of the present invention;
[0054] Figure 2 A flow chart of a method for detecting reduplication errors provided in Embodiment 1 of the present invention;
[0055] Figure 3 Schematic diagram of some effects of the method for detecting reduplication errors provided in the first embodiment of the present invention Figure 1 ;
[0056] Figure 4 Schematic diagram of some effects of the method for detecting reduplication errors provided in the first embodiment of the present invention Figure 2 ;
[0057] Figure 5 A schematic diagram of the structure of a device for detecting reduplication errors provided in Embodiment 2 of the present invention;
[0058] Figure 6 A schematic diagram of the structure of a system for detecting reduplication errors provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0060] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0061] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.
[0062] Figure 1 A schematic diagram of a typical application scenario of the present invention is shown in FIG. Figure 1 As shown, the system 11 for detecting reduplication errors of the present invention obtains the text, for example, we analyze this case, obtain multiple texts to be detected, and obtain the repeated phrases in the text to be detected; the original text segment where the repeated phrase is located is segmented to detect whether there are adjacent reduplications; if there are adjacent reduplications, the part of speech of the adjacent reduplications is matched in the preset part of speech database; if the match is successful, a reduplication in the original text segment is deleted to obtain the text segment to be detected, and the use probability of the original text segment and the text segment to be detected is obtained; if the use probability of the original text segment is less than the use probability of the text segment to be detected, it is determined that the adjacent reduplication is used incorrectly in the original text segment. In an optional embodiment, the adjacent reduplication that is used incorrectly can also be marked (e.g., bolded) in the text. To realize the judgment of the misuse of reduplication, improve the accuracy and speed of detection, and can adapt to sentences of different lengths, thereby improving the accuracy of the judgment of reduplication in long sentences.
[0063] Figure 2The flowchart of the method for detecting reduplicated word errors provided in the first embodiment of the present invention is as follows. Figure 2 As shown, the method for detecting reduplicated word errors in this embodiment may include:
[0064] S201. Obtain a text, and obtain the punctuation marks and string information in the text.
[0065] In this embodiment, the system for detecting reduplicated word errors can obtain the text by directly downloading or copying an electronic text from the network, or by scanning a paper text and converting it into an electronic text in the required format. There is no specific limitation in this embodiment, and it can be further specifically limited according to actual needs to achieve better results. The text includes, but is not limited to, Chinese characters, punctuation marks, and strings, etc.
[0066] Find the positions that do not require reduplicated word detection from the text according to string and rule matching. For example: non-Chinese character characters or strings, various punctuation marks (such as commas, periods, etc.), especially including the content within the book title marks 《xxx》, the content within the quotation marks “yyy”, the content within the parentheses (zzz), etc.
[0067] S202. Obtain multiple texts to be detected according to the punctuation marks and string information, and obtain the repeated phrases in the texts to be detected.
[0068] In this embodiment, according to the positions of non-Chinese characters in the above example text, the text is divided into multiple texts to be detected. The text to be detected is all or part of the text fragment of this text. For example, for the text: I am reading the book 《Autumn's xx》, which tells the story of a little girl walking in the woods in late autumn, and the fiery red maple leaves are falling from the trees. This text is divided into: I am reading, this book, telling the story of a little girl walking in the woods in late autumn, and the fiery red maple leaves are falling from the trees, etc. four texts to be detected. Then, obtain the repeated phrases from the texts to be detected. In an optional embodiment, the repeated phrases can be adjacent or non-adjacent. For example, in the above example, the adjacent repeated phrase “fiery red” is obtained from the text to be detected “the fiery red maple leaves are falling from the trees”.
[0069] S203. Segment the original text fragment where the repeated phrase is located, and detect whether there are adjacent reduplicated words.
[0070] Specifically, Chinese reduplicated words are words composed of two or more Chinese characters with the same glyph and meaning, overlapping and used together, and can include reduplicated words in forms such as AAB, ABB, AA, ABA, ABAB, AABB, etc. For example, they are respectively “give a hand”, “laughing笑呵呵”, “often往往”, “from time to time时不时”, “呼啦呼啦hula hula”, “quiet安安静静”, etc.
[0071] Combined with the above example, segment the original text fragment where adjacent repeated phrases are located. For example, obtain "fiery red / fiery red / of / maple leaves / falling / from / the tree", and then detect the adjacent reduplicated word "fiery red".
[0072] S204. If there are adjacent reduplicated words, match the part-of-speech of the adjacent reduplicated words in the preset part-of-speech database.
[0073] In an optional embodiment, matching the part-of-speech of the adjacent reduplicated words in the preset part-of-speech database includes:
[0074] Perform part-of-speech tagging on the adjacent reduplicated words, and match the part-of-speech of the adjacent reduplicated words with the pre-stored part-of-speech in the preset part-of-speech database.
[0075] Specifically, in an optional embodiment, before matching the adjacent reduplicated words in the preset part-of-speech database, obtain a large number of reduplicated words, perform part-of-speech tagging on these reduplicated words. For example, tag the part-of-speech of the reduplicated word "quietly" as an adjective, and store the corresponding part-of-speech data of all the obtained reduplicated words in the preset part-of-speech database for subsequent part-of-speech matching of the reduplicated words to be detected.
[0076] In an optional embodiment, it further includes:
[0077] If there is no pre-stored part-of-speech in the preset part-of-speech database that matches the adjacent reduplicated words, determine that the adjacent reduplicated words are used incorrectly.
[0078] For example, tag the part-of-speech of the existing adjacent reduplicated word "analyze analyze" as a verb and perform matching in the preset part-of-speech database. However, if there is no pre-stored part-of-speech in the preset part-of-speech database that matches this adjacent reduplicated word, determine that this adjacent reduplicated word is used incorrectly.
[0079] S205. If the matching is successful, delete one reduplicated word in the original text fragment to obtain the text fragment to be detected, and obtain the usage probabilities of the original text fragment and the text fragment to be detected.
[0080] In an optional embodiment, obtaining the usage probabilities of the original text fragment and the text fragment to be detected includes:
[0081] Use a moving window to move respectively on the original text fragment and the text fragment to be detected to obtain multiple target test characters excluding the reduplicated words;
[0082] After detecting the usage probabilities of the target test characters in the original text fragment and the text fragment to be detected respectively through the preset language model, determine the usage probabilities of the original text fragment and the text fragment to be detected.
[0083] Specifically, when there is a pre-stored part of speech matching the adjacent reduplication in the preset part of speech database, a reduplication in the original text segment is deleted to obtain the text segment to be detected. For example, a reduplication "analysis" in the original text segment "We analyzed this case." is deleted to obtain the text segment to be detected "We analyzed this case."
[0084] Then, the usage probabilities of the original text segment and the text segment to be detected are obtained, including: using a moving window to move on the original text segment and the text segment to be detected respectively, for example, the original text segment is "We analyzed this case.", and the text segment to be detected is "We analyzed this case." Reference Figure 3 , Figure 3 Schematic diagram of some effects of the method for detecting reduplication errors provided in the first embodiment of the present invention Figure 1 ,like Figure 3 As shown, the moving window starts from the position of the reduplication "analysis" or "analysis analysis", and moves to the two end positions in turn to obtain multiple target test characters without reduplication, such as "we" and so on. Then, through the preset language model, after detecting the usage probability of the target test character such as "we" in the original text segment and the text segment to be detected, the usage probability of the original text segment and the text segment to be detected is determined. The preset language model can learn the rules of the Chinese language after being trained with large-scale Chinese corpus. After removing the reduplication at a certain position in the text segment, the preset language model can be used to obtain the probability of using the reduplication at that position in the text segment. In an optional embodiment, the formula is used. ,and In an optional embodiment, the pre-trained language model may be a Chinese language model trained by models such as Bert, Ernie, and XLNet.
[0085] Then, according to the usage probabilities of all the target test words in the original text segment and the text segment to be tested, the usage probabilities of the original text segment and the text segment to be tested are determined. Assuming that the words in the context windows of the original sentence and the comparison sentence are x1, x2, x3, and x4 respectively, using the pre-trained language model, the calculated probabilities are .if , then the original text fragment has a higher probability of being used and the reduplication is the correct usage, otherwise the reduplication is a misuse.
[0086] In an optional embodiment, it also includes:
[0087] According to the original text segment and the text segment to be detected, set the parameters of the moving window;
[0088] According to the size parameter of the moving window, the moving window starts from the positions of the reduplications in the original text segment and the text segment to be detected, and moves to the positions at both ends in turn to obtain multiple target inspection characters without reduplications.
[0089] Specifically, the moving window may include parameter settings corresponding to the border, parameter settings corresponding to the size, and parameter settings corresponding to the color. The parameters of the moving window are set according to the original text segment and the text segment to be detected, especially the size parameters of the moving window. As the size of the moving window is larger, the accuracy is relatively higher, but the moving speed of the moving window will also increase exponentially. Therefore, according to the specific settings in the actual detection according to the original text segment and the text segment to be detected, when the size value n corresponding to the moving window in this embodiment is 2, a higher accuracy can be obtained.
[0090] For example, the moving window starts from the positions of the reduplications in the original text segment and the text segment to be detected, and moves to the positions at both ends in turn to obtain multiple target test characters without reduplications. Figure 4 The target test words obtained in the test, such as "I", "the sky is full of hair", "snowflakes blooming", etc., Figure 4 Schematic diagram of some effects of the method for detecting reduplication errors provided in the first embodiment of the present invention Figure 2 .
[0091] In an optional embodiment, it also includes:
[0092] If the target inspection word does not fill the moving window, the moving window is determined to be at a partial position at one end of the original text segment and the text segment to be detected, and the moving window is moved at a partial position at the other end to fill the moving window with the target inspection word.
[0093] Specifically, the moving window takes the first n characters of the reduplication and the last n characters (including punctuation marks) as the target test characters. If the first character is less than n, the length of the following window will be increased accordingly; conversely, if the last character is less than n, the length of the first window will be increased accordingly.
[0094] refer to Figure 4 For example, n=2, when moving the moving window on the original text segment and the text segment to be detected, e.g. Figure 4In the examples corresponding to the second, third and fourth rows in , no matter which case, the position of one end portion is determined respectively when moving the moving window. If the detected target test word does not fill the moving window, then when the moving window is moved at the other end portion of the original text segment or the text segment to be detected, the target test word needs to fill the moving window. For example, if there is only one target test word corresponding to the partial position of one end, then three target test words are required to correspond to the partial position of the other end; or if the test word corresponding to the partial position of one end is empty, then four target test words are required to correspond to the partial position of the other end; or no matter which part of the target test word corresponding to the partial position does not meet the preset number of target test words, for example, the difference of the missing number is 1, then the target test word corresponding to the other end position is used to increase the target test word with the corresponding difference of 1, wherein the target test word also includes punctuation marks.
[0095] S206: If the usage probability of the original text segment is less than the usage probability of the text segment to be detected, it is determined that the adjacent reduplicated words are used incorrectly in the original text segment.
[0096] Combined with the above example, assuming that the characters in the original text segment and the moving windows of the context in the text segment to be detected are x1, x2, x3, and x4 respectively, using the preset language model, the calculated probabilities are .if , then the original text fragment has a higher probability of being used and the reduplication is the correct usage, otherwise the reduplication is a misuse.
[0097] For example, "We analyzed this data and found many problems." Inspection result: "Analysis" is used repeatedly, which is a reduplication error. For another example, "This is a classic case. Let's sit down and analyze it together." Inspection result: "Analysis" here does not belong to a reduplication error. For another example, "The website is polarized. Judging from the advertisements on the homepage, it is mainly aimed at middle-aged and elderly people." Inspection result: "Mainly" is used repeatedly, which is a reduplication error. For another example, "Xibei's rapid development has enabled it to unconsciously extend its business to various business opportunities it encounters." Inspection result: "To" is used repeatedly, which is a reduplication error. For another example, "Autumn is here, and the fiery red maple leaves fall from the trees." Inspection result: "Fiery red" here does not belong to a reduplication error. In an optional embodiment, the preset language model can use a Chinese language model trained by models such as Bert, Ernie, and XLNet.
[0098] In an optional embodiment, after determining that adjacent reduplication words are used incorrectly in the original text segment, the method further includes:
[0099] Marks incorrect use of adjacent reduplications in text.
[0100] For example, the original text segment is "I also analyzed the important parts thereof". The method of the above example can not only accurately detect the incorrect use of reduplication, but also mark the incorrectly used adjacent reduplications in the corresponding text "I read a lot of information and also analyzed the important parts thereof" for subsequent use, such as marking them in bold, which is not limited to this embodiment.
[0101] Figure 5 A schematic diagram of the structure of the device for detecting reduplication errors provided in the second embodiment of the present invention is shown in FIG. Figure 5 As shown, the device for detecting reduplication errors in this embodiment may include:
[0102] The acquisition module 31 is used to acquire text and obtain punctuation marks and character string information in the text;
[0103] An obtaining module 32 is used to obtain a plurality of texts to be detected according to punctuation marks and character string information, and obtain repeated phrases in the texts to be detected;
[0104] A detection module 33 is used to segment the original text segment where the repeated phrase is located to detect whether there are adjacent reduplications;
[0105] A matching module 34 is used to match the parts of speech of adjacent reduplication words in a preset part of speech database if there are adjacent reduplication words;
[0106] A deletion module 35 is used to delete a redundant word in the original text segment to obtain the text segment to be detected if the match is successful, and obtain the usage probability of the original text segment and the text segment to be detected;
[0107] The determination module 36 is used to determine that the adjacent reduplicated words are used incorrectly in the original text segment if the usage probability of the original text segment is less than the usage probability of the text segment to be detected.
[0108] In a possible design, the usage probabilities of the original text segment and the text segment to be detected are obtained, including:
[0109] A moving window is used to move on the original text segment and the text segment to be detected respectively to obtain multiple target test characters without reduplication;
[0110] By using a preset language model, the usage probabilities of the target test word in the original text segment and the text segment to be detected are detected respectively, and then the usage probabilities of the original text segment and the text segment to be detected are determined.
[0111] In one possible design, it also includes:
[0112] According to the original text segment and the text segment to be detected, set the parameters of the moving window;
[0113] According to the size parameter of the moving window, the moving window starts from the positions of the reduplications in the original text segment and the text segment to be detected, and moves to the positions at both ends in turn to obtain multiple target inspection characters without reduplications.
[0114] In one possible design, it also includes:
[0115] If the target inspection word does not fill the moving window, the moving window is determined to be at a partial position at one end of the original text segment and the text segment to be detected, and the moving window is moved at a partial position at the other end to fill the moving window with the target inspection word.
[0116] In a possible design, the parts of speech of adjacent reduplicated words are matched in a preset part of speech database, including:
[0117] The adjacent reduplication words are tagged with parts of speech, and the parts of speech of the adjacent reduplication words are matched with the pre-stored parts of speech in the preset part-of-speech database.
[0118] In one possible design, it also includes:
[0119] If there is no pre-stored part of speech matching the adjacent reduplication in the preset part of speech database, it is determined that the adjacent reduplication is used incorrectly.
[0120] In a possible design, after determining that adjacent reduplications are used incorrectly in the original text segment, the following is further included:
[0121] Marks incorrect use of adjacent reduplications in text.
[0122] The device for detecting reduplication errors of this embodiment can execute Figure 2 For the technical solution in the method shown, its specific implementation process and technical principles can be found in Figure 2 The relevant descriptions in the method shown will not be repeated here.
[0123] Figure 6 The schematic diagram of the structure of the system for detecting reduplication errors provided by the third embodiment of the present invention is as follows: Figure 6 As shown, the system 40 for detecting reduplication errors of this embodiment may include: a processor 41 and a memory 42.
[0124] A memory 42, for storing computer programs (such as application programs, function modules, etc. for implementing the above-mentioned method for detecting reduplication errors), computer instructions, etc.;
[0125] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories 42 in partitions. And the above-mentioned computer programs, computer instructions, data, etc. may be called by the processor 41.
[0126] The processor 41 is used to execute the computer program stored in the memory 42 to implement the various steps in the method involved in the above embodiment.
[0127] For details, please refer to the relevant description in the previous method embodiment.
[0128] The processor 41 and the memory 42 may be independent structures or integrated structures. When the processor 41 and the memory 42 are independent structures, the memory 42 and the processor 41 may be coupled and connected via a bus 43 .
[0129] The server of this embodiment can execute Figure 2 For the technical solution in the method shown, its specific implementation process and technical principles can be found in Figure 2 The relevant descriptions in the method shown will not be repeated here.
[0130] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When at least one processor of a user device executes the computer-executable instructions, the user device executes the above-mentioned various possible methods.
[0131] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.
[0132] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for detecting reduplication errors, characterized in that: include: Acquire text, and obtain punctuation marks and character string information in the text; Obtaining a plurality of texts to be detected according to the punctuation marks and the character string information, and obtaining repeated phrases in the texts to be detected; Segmenting the original text segment containing the repeated phrase to detect whether there are adjacent reduplications; If the adjacent reduplication exists, matching the part of speech of the adjacent reduplication in a preset part of speech database; If the match is successful, a reduplication in the original text segment is deleted to obtain a text segment to be detected, and the parameters of the moving window are set according to the original text segment and the text segment to be detected; according to the size parameters of the moving window, the moving window starts from the positions of the reduplication in the original text segment and the text segment to be detected, and moves to the positions of both ends in sequence to obtain a plurality of target inspection characters without the reduplication; After respectively detecting the usage probabilities of the target test word in the original text segment and the text segment to be detected by using a preset language model, the usage probabilities of the original text segment and the text segment to be detected are determined; If the usage probability of the original text segment is less than the usage probability of the text segment to be detected, it is determined that the adjacent reduplicated words are used incorrectly in the original text segment.
2. The method according to claim 1, characterized in that Also includes: If the target inspection word does not fill the moving window, the moving window is determined to be at a partial position at one end of the original text segment and the text segment to be detected, and when the moving window is moved at a partial position at the other end, the target inspection word fills the moving window.
3. The method according to claim 1, characterized in that Matching the parts of speech of the adjacent reduplication words in a preset part of speech database includes: Parts of speech are marked for the adjacent reduplications, and the parts of speech of the adjacent reduplications are matched with pre-stored parts of speech in a preset part-of-speech database.
4. The method according to claim 3, characterized in that: Also includes: If there is no pre-stored part of speech matching the adjacent reduplication word in the preset part of speech database, it is determined that the adjacent reduplication word is used incorrectly.
5. The method according to any one of claims 1 to 4, characterized in that: After determining that the adjacent reduplication words are used incorrectly in the original text segment, the method further includes: Marks adjacent reduplications that are used incorrectly in the text.
6. A device for detecting reduplication errors, characterized in that: include: An acquisition module, used to acquire text and obtain punctuation marks and character string information in the text; An obtaining module, used for obtaining a plurality of texts to be detected according to the punctuation marks and the string information, and obtaining repeated phrases in the texts to be detected; A detection module, used to segment the original text segment where the repeated phrase is located, and detect whether there are adjacent reduplications; A matching module, for matching the part of speech of the adjacent reduplication words in a preset part of speech database if the adjacent reduplication words exist; A deletion module is used for deleting a redundant word in the original text segment to obtain a text segment to be detected if the match is successful, and setting the parameters of the moving window according to the original text segment and the text segment to be detected; according to the size parameters of the moving window, the moving window starts from the redundant word position in the original text segment and the text segment to be detected, and moves to the positions of both ends in sequence to obtain a plurality of target inspection characters without the redundant word; After respectively detecting the usage probabilities of the target test word in the original text segment and the text segment to be detected by using a preset language model, the usage probabilities of the original text segment and the text segment to be detected are determined; The determination module is used to determine that the adjacent reduplicated words are used incorrectly in the original text segment if the usage probability of the original text segment is less than the usage probability of the text segment to be detected.
7. A system for detecting reduplication errors, characterized in that: include: A memory and a processor, wherein the memory stores executable instructions of the processor; wherein the processor is configured to execute the method for detecting reduplication errors described in any one of claims 1-5 by executing the executable instructions.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for detecting reduplication errors described in any one of claims 1 to 5 is implemented.
9. A computer program product comprising: A computer program, characterized in that when the computer program is executed by a processor, it is used to implement the method for detecting reduplication errors described in any one of claims 1-5.
