Method and apparatus for determining target focus words to be corrected in text

Abstract

The purpose of the present application is to provide a method and device for determining a target key word to be corrected in text. Compared with the prior art, the present application acquires a text to be corrected and a target key word, performs n-gram full-text matching in the text to be corrected based on the target key word, determines a matched n-gram unit, wherein n is a preset natural number not greater than the length of the target key word, then merges n-gram units with a distance less than a preset threshold, determines a set of merged candidate sub-strings, determines a sub-string with a length meeting a threshold from the set of candidate sub-strings as a target candidate sub-string, finally calculates the similarity between the target candidate sub-string and the target key word, and corrects the target key word with a similarity greater than a similarity threshold. In this way, it does not need to rely on training samples, has strong generalization ability and high flexibility, and can adapt to various task scenarios.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a technique for identifying target key words to be corrected in text. Background Technology

[0002] In existing technologies, common error correction schemes consider implementation using statistical language models and deep learning. For example, statistical language model methods detect errors in the target character and sort the confusion set to select the top candidate words to correct erroneous text. Another example is neural network-based error correction using a seq2seq model. This involves inputting a sentence into the encoder, processing it, and then having the decoder output the character with the top confidence score at each position in the sentence, thus correcting the error.

[0003] However, the above-mentioned existing technologies have the following drawbacks: (1) The statistical model only considers the probability of a single character forming a word, and this probability depends on the training samples, thus limiting the generalization ability. (2) The neural network model only corrects errors based on contextual semantic information, without considering specific task scenarios, and has low flexibility in response to users' changing key word requirements. Summary of the Invention

[0004] The purpose of this application is to provide a method and apparatus for determining target key words to be corrected in text, so as to overcome the problems of limited generalization ability and low flexibility in the prior art.

[0005] According to one aspect of this application, a method is provided for determining target key words to be corrected in text, wherein the method includes: Obtain the text to be corrected and the target keywords; Based on the target key words, n-gram full-text matching is performed on the text to be corrected to determine the matching n-gram units, where n is a preset natural number not greater than the length of the target key words; n-gram units with a distance less than a preset threshold are merged to determine the candidate substring set after merging; From the set of candidate substrings, substrings whose length meets the threshold are selected as target candidate substrings; Calculate the similarity between the target candidate substring and the target key word, and use the target candidate substring with a similarity greater than the similarity threshold as the target key word to be corrected.

[0006] Furthermore, the n-gram full-text matching includes homophonic matching of the first and last words of the target key words.

[0007] Optionally, the step of merging n-gram units with a distance less than a preset threshold to determine the merged candidate substring set includes: n-gram units with a distance less than a preset threshold are completely merged to determine the merged candidate substring set, wherein complete merging means that the candidate substring includes the first and last words of the target key word. Optionally, the method further includes: Determine whether there are words with the same initial and / or final characters as the target key word in adjacent positions of the candidate substring in the text to be corrected; If there is a word with the same initial and / or final character as the target keyword, add the initial and / or final character to the candidate substring.

[0008] Optionally, the method further includes: Candidate substrings with the same beginning but different ending and / or the same ending but different beginning among substrings whose length meets the threshold are compared with the target key words, and the substrings whose similarity reaches the comparison threshold are determined as target candidate substrings.

[0009] According to another aspect of this application, a device for determining target key words to be corrected in text is also provided, wherein the device includes: The first device is used to acquire the text to be corrected and the target keywords; The second device is used to perform n-gram full-text matching in the text to be corrected based on the target key words, and determine the matching n-gram units, where n is a preset natural number not greater than the length of the target key words; The third device is used to merge n-gram units whose distance is less than a preset threshold to determine the candidate substring set after merging; The fourth device is used to determine, from the set of candidate substrings, a substring whose length meets a threshold as a target candidate substring; The fifth device is used to calculate the similarity between the target candidate substring and the target key word, and to correct the target candidate substring with a similarity greater than the similarity threshold as the target key word to be corrected.

[0010] According to another aspect of this application, a computer-readable medium is also provided, on which computer-readable instructions are stored, which can be executed by a processor to perform the operations as described above.

[0011] Compared with existing technologies, this application obtains the text to be corrected and the target keywords, and performs n-gram full-text matching on the text based on the target keywords to determine the matching n-gram units, where n is a preset natural number not greater than the length of the target keywords. Then, n-gram units with a distance less than a preset threshold are merged to determine a merged candidate substring set. Substrings whose length meets the threshold are selected from the candidate substring set as target candidate substrings. Finally, the similarity between the target candidate substrings and the target keywords is calculated, and target candidate substrings with a similarity greater than the similarity threshold are used as target keywords to be corrected. This method does not rely on training samples, has strong generalization ability and high flexibility, and can adapt to various task scenarios. Attached Figure Description

[0012] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart is shown for determining target key words to be corrected in text, according to one aspect of this application; Figure 2 A schematic diagram of a device for determining target key words to be corrected in text, according to another aspect of this application.

[0013] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0014] The present invention will now be described in further detail with reference to the accompanying drawings.

[0015] In a typical configuration of this application, the terminal, the device of the service network, and the trusted party all include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0016] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0017] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0018] To further illustrate the technical means adopted and the effects achieved in this application, the technical solution of this application will be clearly and completely described below in conjunction with the accompanying drawings and preferred embodiments.

[0019] Figure 1 A method for determining target key words to be corrected in text, according to one aspect of this application, is shown. The method is performed on device 1 and includes the following steps: S11 retrieves the text to be corrected and the target keywords; S12 performs n-gram full-text matching on the text to be corrected based on the target key words, and determines the matching n-gram units, where n is a preset natural number not greater than the length of the target key words; S13 merges n-gram units whose distance is less than a preset threshold to determine the candidate substring set after merging; S14 determines the target candidate substring from the candidate substring set whose length meets the threshold. S15 calculates the similarity between the target candidate substring and the target key word, and corrects the target candidate substring with a similarity greater than the similarity threshold as the target key word to be corrected.

[0020] In this application, the device 1 includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud composed of multiple servers; here, the cloud is composed of a large number of computers or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing and consists of a virtual supercomputer composed of a group of loosely coupled computers. The above device 1 is only an example, and other existing or future devices 1 that can be applied to this application should be included within the protection scope of this application and are hereby incorporated by reference.

[0021] In this embodiment, in step S11, the text to be corrected and the target key word are obtained. Here, the target key word includes any word set by the user for correcting the set word in the text to be corrected. Specifically, the text to be corrected includes any text selected by the user or text randomly selected by the device 1 or selected based on a preset rule.

[0022] Continuing in this embodiment, in step S12, n-gram full-text matching is performed on the text to be corrected based on the target key word to determine the matching n-gram units, where n is a preset natural number not greater than the length of the target key word. Here, the setting of n can be arbitrarily set by the user or randomly set by the device 1. For example, generally n can be set to 2. Specifically, the position set of the matching n-gram units can be determined through matching. For example, if the text to be corrected is "We should not only be down-to-earth but also aim high. Rather be down-to-earth than aim high", and the target key word is "teach down-to-earth is aim high", and n is 2, then the 2-gram units obtained after 2-gram matching of the above corrected text are as follows: "footstep, down-to-earth, earth, aim high, high aim, high aim, footstep, down-to-earth, earth, aim high, high aim, high aim, down-to-earth, is aim, aim high, high aim".

[0023] Preferably, the n-gram full-text matching includes homophone and homograph matching of the first and last words of the target key word. In this embodiment, during full-text matching, homophones or homographs of the first and last words of the target key word will also be matched. For example, if the target key word is "down-to-earth is aim high", if there are words in the text to be corrected that are homophones or homographs of "foot" or "earth", they will also be matched, such as "teach down" and "real first", etc. By this means, it is possible to avoid missing target key words with misspelled first and last words.

[0024] Continuing in this embodiment, in step S13, n-gram units with a distance less than a preset threshold are merged to determine the merged candidate substring set. Here, a distance less than the preset threshold refers to n-gram units with a relatively close distance. Here, the preset threshold can be set by the user or determined based on existing empirical values, and no specific limitation is made here. For example, for the text to be corrected "我们既要脚踏实地,也要志存高远。", the obtained 2-gram units include "脚踏, 踏实, 实地, 志存, 存高, 高远", then the obtained candidate substring set includes "脚踏实地, 志存高远".

[0025] Preferably, step S13 includes: completely merging n-gram units with a distance less than a preset threshold to determine the merged candidate substring set, where complete merging means that the candidate substring includes the first and last words of the target key word. In this embodiment, through the way of complete merging, the target candidate substring closest to the target key word can be more accurately determined through the candidate substring.

[0026] Preferably, the method further includes: determining whether there are first words and / or last words homomorphic with the target key word at adjacent positions of the candidate substring in the text to be corrected; when there are first words and / or last words homomorphic with the target key word, adding the first word and / or last word to the candidate substring.

[0027] In this embodiment, the situation of having extra words will also be corrected. For example, if there is a "脚" in front of "脚踏实地", then it is added to the candidate substring to become "脚脚踏实地", so as to correct the situation of having extra words.

[0028] Continuing in this embodiment, in step S14, a substring with a length meeting the threshold is determined from the candidate substring set as the target candidate substring. For example, for the target key word "脚踏实地就是志存高远", the obtained candidate substring set includes "脚踏实地, 志存高远, 脚踏实地, 志存高远, 教踏实第是志存高远", then the first four results will be screened out because the lengths of the first four results do not match the target key word. Therefore, "教踏实第是志存高远" is retained as the final output.

[0029] Continuing in this embodiment, in step S15, the similarity between the target candidate substring and the target key word is calculated, and the target candidate substring with a similarity greater than the similarity threshold is used to correct the target key word to be corrected. Here, the similarity can be presented in the form of a score, and the target candidate substring with a score greater than a certain threshold is used as the target key word to be corrected. Through this method, the determined target candidate substring can be closest to the target key word.

[0030] Preferably, the method further includes: comparing candidate substrings with the same beginning but different ending and / or the same ending but different beginning among substrings whose length meets the threshold with the target key word, and determining the substrings with similarity reaching the comparison threshold as target candidate substrings. In this embodiment, there may also be candidate substrings with the same beginning but different ending and / or the same ending but different beginning, and these candidate substrings are compared with the target key word to determine the more similar ones as target candidate substrings.

[0031] Compared with existing technologies, this application obtains the text to be corrected and the target keywords, and performs n-gram full-text matching on the text based on the target keywords to determine the matching n-gram units, where n is a preset natural number not greater than the length of the target keywords. Then, n-gram units with a distance less than a preset threshold are merged to determine a merged candidate substring set. Substrings whose length meets the threshold are selected from the candidate substring set as target candidate substrings. Finally, the similarity between the target candidate substrings and the target keywords is calculated, and target candidate substrings with a similarity greater than the similarity threshold are used as target keywords to be corrected. This method does not rely on training samples, has strong generalization ability and high flexibility, and can adapt to various task scenarios.

[0032] In addition, such as Figure 2 As shown in the embodiment of this application, a device 1 for determining target key words to be corrected in text is also provided, wherein the device 1 includes: The first device 11 is used to acquire the text to be corrected and the target key words; The second device 12 is used to perform n-gram full-text matching in the text to be corrected based on the target key words, and determine the matching n-gram units, where n is a preset natural number not greater than the length of the target key words; The third device 13 is used to merge n-gram units whose distance is less than a preset threshold to determine the merged candidate substring set; The fourth device 14 is used to determine, from the set of candidate substrings, a substring whose length meets a threshold as a target candidate substring; The fifth device 15 is used to calculate the similarity between the target candidate substring and the target key word, and to correct the target candidate substring with a similarity greater than the similarity threshold as the target key word to be corrected.

[0033] Furthermore, embodiments of this application also provide a computer-readable medium having computer-readable instructions stored thereon, which can be executed by a processor to implement the aforementioned method.

[0034] This application also provides a device for correcting target key words in text, wherein the device includes: One or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform the operations of the aforementioned method.

[0035] For example, when executed, computer-readable instructions cause one or more processors to: acquire the text to be corrected and the target key words; perform n-gram full-text matching on the text to be corrected based on the target key words to determine the matching n-gram units, where n is a preset natural number not greater than the length of the target key words; merge n-gram units with a distance less than a preset threshold to determine a merged candidate substring set; determine substrings with a length that meets the threshold from the candidate substring set as target candidate substrings; calculate the similarity between the target candidate substrings and the target key words, and correct the target candidate substrings with a similarity greater than the similarity threshold as the target key words to be corrected.

[0036] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A method for determining target key words to be corrected in text, wherein, The method includes: Obtain the text to be corrected and the target keywords; Based on the target key words, n-gram full-text matching is performed on the text to be corrected to determine the matching n-gram units, where n is a preset natural number not greater than the length of the target key words. The n-gram full-text matching refers to performing a sliding window operation of size n on the text content according to the characters to form a character fragment sequence of length n. n-gram units with a distance less than a preset threshold are merged to determine the candidate substring set after merging. The n-gram units with a distance less than the preset threshold refer to adjacent n-gram units. Merging includes keeping only one of the repeated characters in adjacent n-gram units. From the set of candidate substrings, substrings whose length meets the threshold are selected as target candidate substrings; Calculate the similarity between the target candidate substring and the target key word, and use the target candidate substring with a similarity greater than the similarity threshold as the key word to be corrected.

2. The method according to claim 1, wherein, The n-gram full-text matching includes homophonic matching of the first and last words of the target key words.

3. The method according to claim 1 or 2, wherein, The step of merging n-gram units with a distance less than a preset threshold to determine the merged candidate substring set includes: n-gram units with a distance less than a preset threshold are completely merged to determine the merged candidate substring set, wherein complete merging means that the candidate substring includes the first and last words of the target key word.

4. The method according to claim 3, wherein, The method further includes: Determine whether there are words with the same initial and / or final characters as the target key word in adjacent positions of the candidate substring in the text to be corrected; If there is a word with the same initial and / or final character as the target keyword, add the initial and / or final character to the candidate substring.

5. The method according to claim 1, wherein, The method further includes: Candidate substrings with the same beginning but different ending and / or the same ending but different beginning among substrings whose length meets the threshold are compared with the target key words, and the substrings whose similarity reaches the comparison threshold are determined as target candidate substrings.

6. A device for determining target key words to be corrected in text, wherein, The device includes: The first device is used to acquire the text to be corrected and the target keywords; The second device is used to perform n-gram full-text matching on the text to be corrected based on the target key words, and determine the matching n-gram units, where n is a preset natural number not greater than the length of the target key words, wherein the n-gram full-text matching refers to performing a sliding window operation of size n on the text content according to the characters to form a character fragment sequence of length n; The third device is used to merge n-gram units whose distance is less than a preset threshold and determine the merged candidate substring set. The n-gram units whose distance is less than the preset threshold refer to adjacent n-gram units. The merging includes keeping only one of the repeated characters in adjacent n-gram units. The fourth device is used to determine, from the set of candidate substrings, a substring whose length meets a threshold as a target candidate substring; The fifth device is used to calculate the similarity between the target candidate substring and the target key word, and to correct the target candidate substring with a similarity greater than the similarity threshold as the key word to be corrected.

7. A computer-readable medium having stored thereon computer-readable instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 5.

8. A device for determining target key words to be corrected in text, wherein, The device includes: One or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform the method as described in any one of claims 1 to 5.

Images