Sentence recognition method and device, storage medium and electronic device

Abstract

The present invention discloses a sentence recognition method and device, a storage medium, and an electronic device, which can be applied to smart transportation scenarios. The method comprises: determining a set of characters to be corrected in an original sentence; correcting the set of characters to be corrected in the original sentence to obtain a corrected sentence; obtaining the perplexity of the original sentence and the perplexity of the corrected sentence; and determining that the original sentence is incoherent if the perplexity of the corrected sentence decreases by a ratio greater than a preset threshold relative to the perplexity of the original sentence. This invention solves the technical problem of low sentence recognition efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a sentence recognition method and device, a storage medium, and an electronic device. Background Art

[0002] In related technologies, multimedia information published on online platforms often suffers from incoherent sentences. For example, articles or videos published by self-media often contain spelling errors, extra characters, or omissions, resulting in incoherent sentences in the main text or title. This makes it difficult for users to understand the true meaning and reduces the user experience.

[0003] Reviewing and correcting multimedia information published on online platforms can solve the problem of incoherent titles or body sentences. Currently, articles on online platforms are usually reviewed by dedicated reviewers, but this method requires a lot of manpower and is inefficient.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] Embodiments of the present invention provide a sentence recognition method and apparatus, a storage medium, and an electronic device to at least solve the technical problem of low sentence recognition efficiency.

[0006] According to one aspect of an embodiment of the present invention, a sentence recognition method is provided, comprising: determining a set of characters to be corrected in an original sentence, wherein the set of characters to be corrected includes at least one character in the original sentence; performing correction processing on the set of characters to be corrected in the original sentence to obtain a corrected sentence; obtaining the perplexity of the original sentence and the perplexity of the corrected sentence, wherein the perplexity of the original sentence is a parameter obtained by processing the characters in the original sentence using a language model, and the perplexity of the original sentence is used to indicate the degree of incoherence of the original sentence; the perplexity of the corrected sentence is a parameter obtained by processing the characters in the corrected sentence using the language model, and the perplexity of the corrected sentence is used to indicate the degree of incoherence of the corrected sentence; when a decrease ratio of the perplexity of the corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold, determining that the original sentence is an incoherent sentence.

[0007] Optionally, the correction processing of the set of characters to be corrected in the original sentence to obtain a corrected sentence includes one of the following: when the set of characters to be corrected includes characters to be deleted, deleting the characters to be deleted in the original sentence to obtain the corrected sentence; when the set of characters to be corrected includes characters to be replaced, replacing the characters to be replaced with target words in the original sentence to obtain the corrected sentence; when the set of characters to be corrected includes words to be replaced, replacing the words to be replaced with target words in the original sentence to obtain the corrected sentence.

[0008] Optionally, deleting the characters to be deleted from the original sentence includes: when the original sentence includes N consecutive identical characters, determining N-1 characters among the N characters as the characters to be deleted, wherein N is a positive integer greater than or equal to 2; deleting the N-1 characters from the original sentence; or when the original sentence includes M consecutive identical words, determining characters in M-1 words among the M words as the characters to be deleted, wherein M is a positive integer greater than or equal to 2; deleting the M-1 words from the original sentence; or when the original sentence includes T consecutive characters with the same pronunciation, determining the characters to be deleted from the T consecutive characters with the same pronunciation, wherein T is a positive integer greater than or equal to 2; and deleting the characters to be deleted from the T consecutive characters with the same pronunciation in the original sentence.

[0009] Optionally, determining the character to be deleted from the T consecutive characters with the same pronunciation includes: deleting the kth character from the T consecutive characters with the same pronunciation in the original sentence to obtain a kth corrected sentence, where k is greater than or equal to 1 and less than or equal to T; obtaining the perplexity of the original sentence and the perplexity of the kth corrected sentence; and determining that the kth character is the character to be deleted when a decrease ratio of the perplexity of the kth corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold.

[0010] Optionally, the method further includes: determining repeated characters in the original sentence, wherein the number of the repeated characters in the original sentence is greater than a preset value; and determining the characters to be deleted from the repeated characters in the original sentence.

[0011] Optionally, determining the character to be deleted from the repeated characters in the original sentence includes: deleting the i-th repeated character in the original sentence to obtain the i-th revised sentence, where i is greater than or equal to 1; obtaining the perplexity of the original sentence and the perplexity of the i-th revised sentence; and determining that the i-th repeated character is the character to be deleted when the decrease ratio of the perplexity of the i-th revised sentence relative to the perplexity of the original sentence is greater than the preset threshold.

[0012] Optionally, the method further includes: when the ratio of decrease in the perplexity of the i-th revised sentence relative to the perplexity of the original sentence is less than or equal to the preset threshold, deleting the i+1th repeated character in the original sentence to obtain the i+1th revised sentence; obtaining the perplexity of the original sentence and the perplexity of the i+1th revised sentence; when the ratio of decrease in the perplexity of the i+1th revised sentence relative to the perplexity of the original sentence is greater than a preset threshold, determining that the i+1th repeated character is the character to be deleted.

[0013] Optionally, the method further includes: when the original sentence includes a first uncommon word, deleting the jth character in the first uncommon word in the original sentence to obtain a jth corrected sentence, wherein j is greater than or equal to 1, and the number of times the first uncommon word appears in the corpus is less than a preset value; obtaining the perplexity of the original sentence and the perplexity of the jth corrected sentence; when the decrease ratio of the perplexity of the jth corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold, determining that the jth character is the character to be deleted.

[0014] Optionally, the method further includes: when the ratio of decrease in the perplexity of the jth revised sentence relative to the perplexity of the original sentence is less than or equal to a preset threshold, deleting the j+1th character in the uncommon word in the original sentence to obtain the j+1th revised sentence; obtaining the perplexity of the original sentence and the perplexity of the j+1th revised sentence; when the ratio of decrease in the perplexity of the j+1th revised sentence relative to the perplexity of the original sentence is greater than a preset threshold, determining that the j+1th repeated character is the character to be deleted.

[0015] Optionally, the method further comprises: when the original sentence includes a single character, determining the single character as the character to be replaced, wherein the single character is a character located between two adjacent words in the original sentence.

[0016] Optionally, the method further includes: searching for words with the character to be replaced as the first or last character in the corpus; determining the word with the highest number of occurrences in the corpus among the words found as the target word; replacing the character to be replaced with the target character in the original sentence to obtain the revised sentence, including: replacing the character to be replaced with the target word in the original sentence.

[0017] Optionally, the method further includes: when the original sentence includes a second uncommon word, determining the second uncommon word as the word to be replaced, wherein the number of times the second uncommon word appears in the corpus is less than a preset value, and the target word is a word obtained by sorting the characters in the second uncommon word in reverse order, and the number of times the target word appears in the corpus is greater than or equal to the preset value.

[0018] Optionally, obtaining the perplexity of the original sentence includes: inputting the original sentence into the N-gram language model, dividing the original sentence into multiple character sequences through the N-gram language model, wherein each character sequence includes N adjacent characters in the original sentence, wherein N is a positive integer greater than or equal to 2; determining the conditional probability of each character sequence in the multiple character sequences relative to the previous character sequence as the probability corresponding to each character sequence; determining the probability corresponding to the original sentence through the probability corresponding to each character sequence; and determining the perplexity of the original sentence based on the probability corresponding to the original sentence.

[0019] Optionally, when the decrease ratio of the perplexity of the revised sentence relative to the perplexity of the original sentence is greater than a preset threshold, the original sentence is determined to be an incoherent sentence, including: determining the difference between the perplexity of the original sentence and the perplexity of the revised sentence; determining the decrease ratio to be equal to the ratio of the difference to the perplexity of the original sentence; when the decrease ratio is greater than the preset threshold, determining the original sentence to be an incoherent sentence.

[0020] Optionally, after determining that the original sentence is an incoherent sentence, the method further includes: replacing the original sentence with the revised sentence; or generating corresponding tags for the set of characters to be revised in the original sentence.

[0021] According to another aspect of an embodiment of the present invention, a sentence recognition device is further provided, comprising: a first determination module, configured to determine a set of characters to be corrected in an original sentence, wherein the set of characters to be corrected includes at least one character in the original sentence; a processing module, configured to perform correction processing on the set of characters to be corrected in the original sentence to obtain a corrected sentence; an acquisition module, configured to obtain the perplexity of the original sentence and the perplexity of the corrected sentence, wherein the perplexity of the original sentence is a parameter obtained by processing the characters in the original sentence using a language model, and the perplexity of the original sentence is used to indicate the degree of incoherence of the original sentence; the perplexity of the corrected sentence is a parameter obtained by processing the characters in the corrected sentence using the language model, and the perplexity of the corrected sentence is used to indicate the degree of incoherence of the corrected sentence; and a second determination module, configured to determine that the original sentence is an incoherent sentence when a decrease ratio of the perplexity of the corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold.

[0022] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned statement recognition method when running.

[0023] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above sentence recognition method.

[0024] According to another aspect of an embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the sentence recognition method through the computer program.

[0025] In an embodiment of the present invention, a correction process is performed on a set of characters to be corrected in an original sentence to obtain a corrected sentence. The perplexity of the original and corrected sentences is used to determine whether the original sentence is incoherent. This achieves the goal of computer-assisted sentence recognition and solves the technical problem of low sentence recognition efficiency caused by manual review in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 is a schematic diagram of an application environment of an optional sentence recognition method according to an embodiment of the present invention;

[0028] Figure 2 is a flowchart of an optional sentence recognition method according to an embodiment of the present invention;

[0029] Figure 3 is a flowchart of another optional sentence recognition method according to an embodiment of the present invention;

[0030] Figure 4 is a flowchart of another optional sentence recognition method according to an embodiment of the present invention;

[0031] Figure 5 is a flowchart of another optional sentence recognition method according to an embodiment of the present invention;

[0032] Figure 6 is a flowchart of another optional sentence recognition method according to an embodiment of the present invention;

[0033] Figure 7 is a flowchart of another optional sentence recognition method according to an embodiment of the present invention;

[0034] Figure 8 is a flowchart of another optional sentence recognition method according to an embodiment of the present invention;

[0035] Figure 9 is a schematic diagram of an optional interface display according to an embodiment of the present invention;

[0036] Figure 10 is another optional interface display schematic diagram according to an embodiment of the present invention;

[0037] Figure 11 This is an optional overall framework diagram according to an embodiment of the present invention;

[0038] Figure 12 is a schematic structural diagram of an optional sentence recognition device according to an embodiment of the present invention;

[0039] Figure 13 is a schematic structural diagram of an optional electronic device according to an embodiment of the present invention;

[0040] Figure 14 This is a block diagram of a computer system structure of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0042] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] According to one aspect of an embodiment of the present invention, a sentence recognition method is provided. Optionally, as an optional implementation, the sentence recognition method can be applied to, but is not limited to, Figure 1 The application environment shown in FIG. The application environment includes a terminal device 102, a network 110, and a server 112. The terminal device includes a memory 104, a processor 106, and a display 108. The memory 104 is used to store data, including but not limited to the original sentence and the revised sentence. The processor 106 is used to process the data, including but not limited to determining a set of characters to be revised in the original sentence and revising the set of characters to be revised in the original sentence. The display 108 is used to display the original sentence.

[0044] Optionally, in this embodiment, the terminal device may be a terminal device configured with a target client, and may include, but is not limited to, at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Device), a PAD, a desktop computer, a smart TV, etc. The target client may be a video client, an instant messaging client, a browser client, a game client, etc.

[0045] Optionally, the network 110 may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that implement wireless communication.

[0046] Optionally, the server 112 includes a database 114 and a processing engine 116. The database is used to store data, including but not limited to storing the original statement and the revised statement. The processing engine is used to process the data, including but not limited to performing the following steps:

[0047] Step S102, determining a character set to be corrected in the original sentence, wherein the character set to be corrected includes at least one character in the original sentence;

[0048] Step S104, performing correction processing on the set of characters to be corrected in the original sentence to obtain a corrected sentence;

[0049] Step S106: Obtain the perplexity of the original sentence and the perplexity of the revised sentence, wherein the perplexity of the original sentence is a parameter obtained by processing characters in the original sentence using a language model, and the perplexity of the original sentence is used to indicate the degree of incoherence of the original sentence; the perplexity of the revised sentence is a parameter obtained by processing characters in the revised sentence using the language model, and the perplexity of the revised sentence is used to indicate the degree of incoherence of the revised sentence;

[0050] Step S108 , when the decrease ratio of the perplexity of the modified sentence relative to the perplexity of the original sentence is greater than a preset threshold, the original sentence is determined to be an incoherent sentence.

[0051] The server may be a single server, a server cluster composed of multiple servers, or a cloud server. The above is only an example and is not limited in this embodiment.

[0052] Optionally, the above sentence recognition method can be applied to intelligent transportation systems. For example, it can be applied to intelligent vehicle terminals. Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing, strengthening the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment, and saves energy.

[0053] Optionally, as an alternative implementation, as Figure 2 shown, the above statement recognition method includes:

[0054] Step S202, determine a set of characters to be corrected in the original statement, where the set of characters to be corrected includes at least one character in the original statement;

[0055] Among them, the above original statement includes, but is not limited to, statements in articles published on network platforms, or titles of videos, audios, etc. The set of characters to be corrected can be one character or a combination of multiple characters. For example, "度" in "爸爸度陷入自责,深深思念女儿" is a character to be corrected, and "画面看" in "脑力测试:仔细画面看中,有几处不对劲" is a set of characters to be corrected, which includes three characters to be corrected, namely "画", "面", and "看". <00001....​​​​​​​​​​The language model can be an N-gram language model, which can be trained based on a large amount of unsupervised corpus. The N-gram language model is an algorithm based on a statistical language model. This model assumes that the occurrence of the Nth word in a sentence is only related to the N-1 preceding words; that is, the probability of the Nth word's occurrence is related to the N-1 preceding words. The probability of the entire sentence is the product of the probabilities of each word's occurrence, which can be calculated by counting the number of word occurrences in the corpus. The N-gram language model can be used to assess the rationality of a sentence, that is, its coherence. Language models typically use the perplexity (PPL) to measure the incoherence of a sentence. A higher sentence probability indicates a lower PPL, while a lower sentence probability indicates a higher PPL. In incoherent scenarios, fluent sentences often have a higher PPL, while incoherent sentences often have a lower PPL. Therefore, training an N-gram language model with a large amount of unsupervised corpus and calculating the PPL can be used to measure the incoherence of a sentence.

[0060] Step S208: When the decrease ratio of the perplexity of the modified sentence relative to the perplexity of the original sentence is greater than a preset threshold, the original sentence is determined to be an incoherent sentence.

[0061] If only the PPL calculated by the language model is used to measure whether a sentence is incoherent, a large number of misjudgments will be caused. There are two main reasons for this: the errors of extra or missing words in the incoherent type often only appear at a certain position in the sentence, while the language model calculates the PPL of the entire sentence. It is possible that even if there are more or fewer words, the calculated PPL value is still small, that is, the PPL of the incoherent sentence is small. Because N-gram learns the fluency of a sentence through a large amount of unsupervised corpus, if a new word does not appear in the corpus or appears very few times, even if the sentence itself is fluent, it will cause the PPL value calculated by the N-gram language model to be too high, that is, the PPL of the fluent sentence is large. In this embodiment, the above-mentioned misjudgment can be avoided by judging whether the original sentence is an incoherent sentence based on the decrease ratio of the perplexity of the revised sentence relative to the perplexity of the original sentence.

[0062] Optionally, the correction processing of the set of characters to be corrected in the original sentence to obtain a corrected sentence includes one of the following: when the set of characters to be corrected includes characters to be deleted, deleting the characters to be deleted in the original sentence to obtain the corrected sentence; when the set of characters to be corrected includes characters to be replaced, replacing the characters to be replaced with target words in the original sentence to obtain the corrected sentence; when the set of characters to be corrected includes words to be replaced, replacing the words to be replaced with target words in the original sentence to obtain the corrected sentence.

[0063] As an optional implementation, the characters to be deleted include, but are not limited to, the redundant characters or words in the original statement. For example, in the original statement "Unexpectedly, he even threw it away", the word "even" is a redundant character in the original statement, and "even" is the character to be deleted in the above original statement. After deleting the redundant character "even" in the original statement, the corrected statement "Unexpectedly, he threw it away" is obtained. Or, for example, "A girl in a certain place has her body darkened, her beauty value is lowered, and the girl doesn't even dare to go out", where the set of characters to be corrected is "beauty value beauty value", with a reduplicated word. One of the "beauty value" is a redundant word in the original statement, and "beauty value" is the character to be deleted in the above original statement. After deleting the redundant "beauty value" in the original statement, the corrected statement "A girl in a certain place has her body darkened, her beauty value is lowered, and the girl doesn't even dare to go out" is obtained.

[0064] As an optional implementation, the characters to be replaced include, but are not limited to, characters that lack characters to form words. For example, the "du" in the original statement "Dad du fell into self-blame and deeply missed his daughter" is a redundant character. "Du" can find "once" in the common word list. In the original statement of this example, "du" is the character to be replaced, and the target word is "once". If the "du" in the original statement is replaced with "once", the corrected statement "Dad once fell into self-blame and deeply missed his daughter" is obtained, and the sentence will be more smooth.

[0065] As an optional implementation, the characters to be replaced include, but are not limited to, character combinations with reversed word orders. For example, in "The girl doesn't even dare to du men out", the word order of "du men" is reversed and is the word to be replaced. After replacing the "du men" in the original statement with the target word "men du", the corrected statement "The girl doesn't even dare to men du out" is obtained.

[0066] As an optional implementation, as Figure 3 shown in the flowchart includes the following steps:

[0067] Step S301, obtain the original statement. Among them, the original statement can be obtained from the user terminal. The original statement can be the content of an article edited by the user on the user terminal, or the title of a video, audio, etc.;

[0068] Step S302, perform word segmentation on the original statement. Specifically, an N-gram language model can be used to perform word segmentation on the original statement, or other methods in the prior art can be used to perform word segmentation on the original statement;

[0069] Step S303 determines whether the original sentence contains any characters to be deleted, replaced, or replaced. If the original sentence contains any characters to be deleted, the process proceeds to step S304. If the original sentence contains any characters to be replaced, the process proceeds to step S305. If the original sentence contains any characters to be replaced, the process proceeds to step S306. If the original sentence contains no characters to be deleted, replaced, or replaced, the process proceeds to step S301.

[0070] Specifically, the judgment can be made by a judgment module, wherein the judgment module includes but is not limited to a reduplicated sound judgment module, a reduplicated word judgment module, a reduplicated word judgment module, a missing character judgment module, a word order reversal judgment module, and the like. Assuming that the reduplicated sound judgment module determines that there are continuous reduplicated sound characters in the original sentence, then there are characters to be deleted in the reduplicated sound characters. If the reduplicated word judgment module determines that there are continuous reduplicated words in the original sentence, then there are characters to be deleted in the reduplicated words. If the missing character judgment module determines that there are characters with missing characters in the original sentence, then the characters with missing characters are the characters to be replaced. If the word order reversal judgment module determines that there are words with reversed word order in the original sentence, then the words with reversed word order are the words to be replaced.

[0071] Step S304: If there are characters to be deleted in the original sentence, the characters to be deleted are deleted from the original sentence to obtain a revised sentence;

[0072] Step S305: If the characters to be replaced exist in the original sentence, the characters to be replaced are replaced with the target words in the original sentence to obtain a revised sentence;

[0073] Step S306: If the word to be replaced exists in the original sentence, the word to be replaced is replaced with the target word in the original sentence to obtain a revised sentence;

[0074] Step S307 determines whether the perplexity of the modified sentence relative to the perplexity of the original sentence is greater than a preset threshold. The threshold can be set based on the actual situation, for example, 0.5, 0.4, 0.56, etc. If so, proceed to step S308; otherwise, proceed to step S303.

[0075] Step S308: Determine whether the original sentence is an incoherent sentence.

[0076] Optionally, deleting the characters to be deleted from the original sentence includes: when the original sentence includes N consecutive identical characters, determining N-1 characters among the N characters as the characters to be deleted, wherein N is a positive integer greater than or equal to 2; deleting the N-1 characters from the original sentence; or when the original sentence includes M consecutive identical words, determining characters in M-1 words among the M words as the characters to be deleted, wherein M is a positive integer greater than or equal to 2; deleting the M-1 words from the original sentence; or when the original sentence includes T consecutive characters with the same pronunciation, determining the characters to be deleted from the T consecutive characters with the same pronunciation, wherein T is a positive integer greater than or equal to 2; and deleting the characters to be deleted from the T consecutive characters with the same pronunciation in the original sentence.

[0077] As an optional implementation, the reduplication module can determine whether there is reduplication in the original sentence. The reduplication module performs a judgment operation on each character in the original sentence. When performing the judgment operation on each character, each character on which the judgment operation is performed is called the current character, such as Figure 4 The flowchart shown includes the following steps:

[0078] Step S401, obtaining an original sentence, and performing a judgment operation on each character in the original sentence, each character during the judgment operation is referred to as a current character;

[0079] Step S402, obtaining the i-th character in the original sentence as the current character, where i is greater than or equal to 1;

[0080] Step S403, determine whether the current character is the same as the character before the current character, if so, execute step S404, or determine whether the current character is the same as the character after the current character, if so, execute step S406. If not, execute step S410;

[0081] Step S404: the current character and the previous character are reduplicated characters;

[0082] Step S405, delete the current character or the previous character, obtain a correction statement, and continue to execute S408;

[0083] Step S406: the current character and the next character are reduplicated characters;

[0084] Step S407, delete the current character or the next character to obtain a corrected statement, and proceed to step S408;

[0085] Step S408: Determine whether the decrease ratio of the perplexity of the corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold. If so, execute Step S409; if not, execute Step S410;

[0086] Step S409: The original sentence is an ungrammatical sentence;

[0087] Step S410: Obtain the (i + 1)-th character as the current character, and continue to execute Step S403.

[0088] As an optional implementation, assume the original sentence is "A word to the guys, don't be too promiscuous, or the ending will be something you never expected". The reduplication module determines that "you you" in the original sentence is reduplication. After removing "you", the PPL reduction ratio exceeds the threshold, and it is determined that the original sentence is ungrammatical.

[0089] As an optional implementation, the reduplication module can determine whether there is reduplication in the original sentence. The reduplication module performs a judgment operation on each word in the original sentence. When performing the judgment operation on each word, each word on which the judgment operation is performed is called the current word. As Figure 5 shown in the following steps of the flowchart:

[0090] Step S501: Obtain the original sentence, perform word segmentation on the original sentence, and perform a judgment operation on each word in the original sentence. Each word character when performing the judgment operation is called the current word;

[0091] Step S502: Obtain the i-th word in the original sentence as the current word, where i is greater than or equal to 1;

[0092] Step S503: Determine whether the current word is the same as the previous word of the current word. If it is the same as the previous word, execute Step S504; or determine whether the current word is the same as the next word of the current word. If it is the same as the next word, execute Step S506. If both are not the same, execute Step S510;

[0093] Step S504: The current word and its previous word are reduplication;

[0094] Step S505: Delete the current word or the previous word to obtain the corrected sentence, and continue to execute S408;

[0095] Step S506: The current word and the next word are reduplication;

[0096] Step S507: Delete the current word or the next word to obtain the corrected sentence, and continue to execute Step S508;

[0097] Step S508: Determine whether the decrease ratio of the perplexity of the corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold. If so, execute Step S509; if not, execute Step S510;

[0098] Step S509, the original statement is ungrammatical;

[0099] Step S510, obtain the (i + 1)-th word as the current word, and continue to execute Step S503.

[0100] As an optional implementation manner, assume the original statement is "Someone's body turns black, and their looks are lowered, not even daring to go out of the door". The reduplicated word module determines that "looks looks" is a reduplicated word. After removing "looks", the reduction ratio of the PPL exceeds the threshold, and it is determined that the original sentence is ungrammatical.

[0101] As an optional implementation manner, the reduplicated sound module can determine whether there are consecutive homophonous characters in the original statement. The reduplicated sound module performs a judgment operation on each character in the original statement. When performing the judgment operation on each character, each character on which the judgment operation is performed is called the current character. As Figure 6 shown in the flowchart includes the following steps:

[0102] Step S601, obtain the original statement, and perform a judgment operation on each character in the original statement. Each character during the execution of the judgment operation is called the current character;

[0103] Step S602, obtain the i-th character in the original statement as the current character, where i is greater than or equal to 1;

[0104] Step S603, determine whether the current character has the same pronunciation as the previous character of the current character. If it has the same pronunciation as the previous character, execute Step S604; if they are all different, execute Step S610. Or determine whether the current character has the same pronunciation as the next character of the current character;

[0105] Step S604, the current character and its previous character are reduplicated sounds;

[0106] Step S605, delete the current character to obtain the first corrected statement;

[0107] Step S606, whether the decrease ratio of the perplexity of the first corrected statement relative to the perplexity of the original statement is greater than the preset threshold. If so, execute Step S609; if not, execute Step S607;

[0108] Step S607, delete the previous character of the current character in the original statement to obtain the second corrected statement;

[0109] Step S608, whether the decrease ratio of the perplexity of the second corrected statement relative to the perplexity of the original statement is greater than the preset threshold. If so, execute Step S609; if not, execute Step S610;

[0110] Step S609, the original statement is ungrammatical;

[0111] Step S610: Obtain the (i + 1)-th character as the current character, and continue to execute Step S603.

[0112] As an optional implementation manner, assume the original statement is "The boy has a sense of conquest in politics and has beaten everyone in the school. He is really arrogant". The reduplicated syllable module determines that "zheng" and "zheng" are reduplicated syllables. After removing "zheng", the PPL reduction ratio exceeds the threshold, and it is determined that the original sentence is ungrammatical.

[0113] Optionally, determining the character to be deleted among the consecutive T characters with the same pronunciation includes: deleting the k-th character among the consecutive T characters with the same pronunciation in the original statement to obtain the k-th corrected statement, where k is greater than or equal to 1 and less than or equal to T; obtaining the perplexity of the original statement and the perplexity of the k-th corrected statement; and determining the k-th character as the character to be deleted when the reduction ratio of the perplexity of the k-th corrected statement relative to the perplexity of the original statement is greater than a preset threshold.

[0114] As an optional implementation manner, the original statement is "Today is really kuai kuai kuai kuai le". The reduplicated syllable module determines that the original statement includes four consecutive characters with the same pronunciation, namely "kuai kuai kuai kuai", and T = 4. Then, starting from the first character of the four consecutive characters with the same pronunciation, delete them one by one, and judge the reduction ratio of the perplexity of the corrected statement obtained by deletion relative to the perplexity of the original statement. If the reduction ratio is greater than the preset threshold, determine that this character is the character to be deleted. Taking the above original statement "Today is really kuai kuai kuai kuai le" as an example, first delete the first character "kuai" in "kuai kuai kuai kuai" in the original statement to obtain the first corrected statement "Today is really kuai kuai kuai le". By calculating the perplexity of the first corrected statement and the perplexity of the original statement, it is found that the reduction ratio is greater than the preset threshold, so "kuai" is the character to be deleted. Continue to delete the second character "kuai" in "kuai kuai kuai kuai" in the original statement to obtain the second corrected statement "Today is really kuai kuai kuai le". By calculating the perplexity of the second corrected statement and the perplexity of the original statement, it is found that the reduction ratio is less than the preset threshold, so "kuai" is not the character to be deleted. Continue to delete the third character "kuai" in "kuai kuai kuai kuai" in the original statement to obtain the third corrected statement "Today is really kuai kuai le". By calculating the perplexity of the third corrected statement and the perplexity of the original statement, it is found that the reduction ratio is greater than the preset threshold, so "kuai" is the character to be deleted. Continue to delete the fourth character "kuai" in "kuai kuai kuai kuai" in the original statement to obtain the fourth corrected statement "Today is really kuai kuai le". By calculating the perplexity of the fourth corrected statement and the perplexity of the original statement, it is found that the reduction ratio is greater than the preset threshold, so "kuai" is the character to be deleted. Thus, it is determined that the character to be deleted is among the four consecutive characters with the same pronunciation in the original statement. After deleting the character to be deleted, the corrected statement "Today is really happy" is obtained.

[0115] Optionally, the method further includes: determining repeated characters in the original statement, where the number of the repeated characters in the original statement is greater than a preset value; determining the characters to be deleted among the repeated characters in the original statement.

[0116] As an optional implementation, the above preset value can be determined according to the actual situation, such as 2, 3, 4, 5, etc. The repeated characters in a short sentence can be judged by an intra-sentence repetition module. For example, in the sentence "ABC is miserably abused by these couples because of being single", since "by" appears multiple times in the short sentence, "by" in the original statement is a repeated character, and the characters to be deleted are determined among the repeated characters in the original statement.

[0117] Optionally, determining the characters to be deleted among the repeated characters in the original statement includes: deleting the i-th repeated character in the original statement to obtain the i-th corrected statement, where i is greater than or equal to 1; obtaining the perplexity of the original statement and the perplexity of the i-th corrected statement; determining the i-th repeated character as the character to be deleted when the decrease ratio of the perplexity of the i-th corrected statement relative to the perplexity of the original statement is greater than the preset threshold.

[0118] As an optional implementation, as Figure 7 shown in the flowchart includes the following steps:

[0119] Step S701, obtaining the repeated characters in the original statement;

[0120] Step S702, performing a deletion operation on each character in the repeated characters, and the character during the deletion operation is called the current character;

[0121] Step S703, taking the i-th character in the repeated characters as the current character;

[0122] Step S704, deleting the current character in the original statement to obtain a corrected statement;

[0123] Step S705, judging whether the decrease ratio of the perplexity of the corrected statement relative to the perplexity of the original statement is greater than the preset threshold. If so, execute Step S706; if not, execute Step S707;

[0124] Step S706, determining that the original statement is an ungrammatical statement;

[0125] Step S707, obtaining the (i + 1)-th character in the repeated characters as the current character, and continue to execute Step S704.

[0126] Suppose the original sentence is "ABC is tortured miserably by these couples while being single", and there are two repeated characters in this original sentence. The first repeated character is the first "by" in the original sentence, and the second repeated character is the second "by" in the original sentence. First, delete the first repeated character in the original sentence to obtain the first corrected sentence "ABC is tortured miserably by these couples while single", and obtain the perplexity of the first corrected sentence and the perplexity of the original sentence. If the decrease ratio is greater than the preset threshold, then the first repeated character is the character to be deleted.

[0127] Optionally, the method further includes: when the decrease ratio of the perplexity of the i-th corrected sentence relative to the perplexity of the original sentence is less than or equal to the preset threshold, delete the (i + 1)-th repeated character in the original sentence to obtain the (i + 1)-th corrected sentence; obtain the perplexity of the original sentence and the perplexity of the (i + 1)-th corrected sentence; when the decrease ratio of the perplexity of the (i + 1)-th corrected sentence relative to the perplexity of the original sentence is greater than the preset threshold, determine the (i + 1)-th repeated character as the character to be deleted.

[0128] As an optional implementation manner, suppose the original sentence is "ABC is tortured miserably by these couples while being single", and there are two repeated characters in this original sentence. The first repeated character is the first "by" in the original sentence, and the second repeated character is the second "by" in the original sentence. First, delete the first repeated character in the original sentence to obtain the first corrected sentence "ABC is tortured miserably by these couples while single", and obtain the perplexity of the first corrected sentence and the perplexity of the original sentence. If the decrease ratio is less than the preset threshold, then the first repeated character is not the character to be deleted. Then continue to delete the second repeated character in the original sentence to obtain the second corrected sentence "ABC is tortured miserably by these couples while single", and obtain the perplexity of the second corrected sentence and the perplexity of the original sentence. If the decrease ratio is greater than the preset threshold, then the second repeated character is the character to be deleted.

[0129] Optionally, the method further includes: when the original sentence includes a first rare word, delete the j-th character in the first rare word in the original sentence to obtain the j-th corrected sentence, where j is greater than or equal to 1, and the number of times the first rare word appears in the corpus is less than the preset value; obtain the perplexity of the original sentence and the perplexity of the j-th corrected sentence; when the decrease ratio of the perplexity of the j-th corrected sentence relative to the perplexity of the original sentence is greater than the preset threshold, determine the j-th character as the character to be deleted.

[0130] As an optional implementation, the above corpus records pre-set corpora. The corpus stores a large-scale unsupervised corpus. Based on the corpus, the frequency of each word appearing in the corpus can be obtained. Words with a frequency less than a preset value in the corpus are regarded as rare words. The above preset value can be determined according to the actual situation, for example, it can be 20, 10, 30, etc. Obtain rare words in the original sentence, delete each character in the rare words. If the decrease ratio of the perplexity of the corrected sentence obtained after deletion relative to the perplexity of the original sentence is greater than a preset threshold, the character in the rare word is a character to be deleted.

[0131] As an optional implementation, as Figure 8 shown in the flowchart includes the following steps:

[0132] Step S801, obtain the first rare word in the original sentence;

[0133] Step S802, perform a deletion operation on each character in the first rare word. The character during the deletion operation is called the current character;

[0134] Step S803, take the j-th character in the first rare word as the current character;

[0135] Step S804, delete the current character in the original sentence to obtain a corrected sentence;

[0136] Step S805, determine whether the decrease ratio of the perplexity of the corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold. If so, execute Step S806; if not, execute Step S807;

[0137] Step S806, determine that the original sentence is an ungrammatical sentence;

[0138] Step S807, obtain the (j + 1)-th character in the first rare word as the current character, and continue to execute Step S804.

[0139] As an optional implementation, assume the original sentence is "I thought the little brother would help the beautiful woman put on her shoes, but unexpectedly he even threw them away". By querying the corpus, the number of times "even" appears in the corpus is less than the preset value. After removing "even", the PPL reduction ratio exceeds the threshold, and it is judged that the original sentence is ungrammatical.

[0140] Optionally, the method further includes: when the decrease ratio of the perplexity of the j-th corrected statement relative to the perplexity of the original statement is less than or equal to a preset threshold, deleting the (j + 1)-th character in the rare word in the original statement to obtain the (j + 1)-th corrected statement; obtaining the perplexity of the original statement and the perplexity of the (j + 1)-th corrected statement; when the decrease ratio of the perplexity of the (j + 1)-th corrected statement relative to the perplexity of the original statement is greater than the preset threshold, determining the (j + 1)-th repeated character as the character to be deleted.

[0141] As an optional implementation manner, assume that the original statement is "Thought the little brother would help the beautiful woman put on her shoes, but unexpectedly he threw it away instead". By querying the corpus, the number of occurrences of "ba geng" in the corpus is less than the preset value. First, delete the first character in "ba geng" to obtain the first corrected statement "Thought the little brother would help the beautiful woman put on her shoes, but unexpectedly he geng it away instead", and judge that the decrease ratio of the perplexity of the corrected statement relative to the perplexity of the original statement is less than the preset threshold, so "ba" is not the character to be deleted. Then continue to delete the second character in "ba geng" to obtain the second corrected statement "Thought the little brother would help the beautiful woman put on her shoes, but unexpectedly he threw it away", and determine that the second character "geng" in the first rare word is the character to be deleted.

[0142] Optionally, the method further includes: when the original statement includes a single character, determining the single character as the character to be replaced, where the single character is the character between two adjacent words in the original statement.

[0143] As an optional implementation manner, the single character in the original statement can be judged by the single-character-in-word judgment module, and the single character is between two adjacent words in the original statement. For example, the "du" in the original statement "Dad du fell into self-blame and deeply missed his daughter" is between "Dad" and "fell into", and "du" is the character to be replaced in the original statement.

[0144] Optionally, the method further includes: searching for words starting with or ending with the character to be replaced in the corpus; determining the word with the highest number of occurrences in the corpus as the target word; replacing the character to be replaced with the target character in the original statement to obtain the corrected statement, including: replacing the character to be replaced with the target word in the original statement.

[0145] As an optional implementation, words starting or ending with a single character can be found in the corpus as target words, and the single character is replaced with the target word to obtain a corrected sentence. For example, for the above original sentence "Dad Du fell into self-blame and deeply missed his daughter", the character "Du" can find "once" in the corpus. After replacing "Du" with "once", the corrected sentence is "Dad once fell into self-blame and deeply missed his daughter". If the reduction ratio of the PPL of the corrected sentence compared to the original sentence exceeds the threshold, it is determined that the original sentence is an ungrammatical sentence.

[0146] Optionally, the method further includes: when the original sentence includes a second rare word, determining the second rare word as the word to be replaced, where the number of occurrences of the second rare word in the corpus is less than a preset value, the target word is the word obtained by sorting the characters in the second rare word in reverse order, and the number of occurrences of the target word in the corpus is greater than or equal to the preset value.

[0147] As an optional implementation, the second rare word in the original sentence can be judged by the word order inversion module. For example, in the original sentence "A person's body turns black, their appearance is lowered, and they don't even dare to go out of the door", the word "door connection" has a frequency of occurrence less than the preset value in the corpus. After reversing the characters in "door connection" to get "connection door", the frequency of occurrence of "connection door" in the corpus is greater than the preset value. The above preset value can be determined according to the actual situation, such as 30, 50, 100, etc. After replacing "door connection" in the original sentence with "connection door", if the reduction ratio of the perplexity of the corrected sentence "A person's body turns black, their appearance is lowered, and they don't even dare to go out of the connection door" compared to the perplexity of the original sentence is greater than the preset threshold, then the original sentence is an ungrammatical sentence.

[0148] Optionally, obtaining the perplexity of the original sentence includes: inputting the original sentence into the N-gram language model, and dividing the original sentence into multiple character sequences by the N-gram language model, where each character sequence includes adjacent N characters in the original sentence, and N is a positive integer greater than or equal to 2; determining the conditional probability of each character sequence relative to the previous character sequence in the multiple character sequences as the probability corresponding to each character sequence; determining the probability corresponding to the original sentence through the probability corresponding to each character sequence; and determining the perplexity of the original sentence according to the probability corresponding to the original sentence.

[0149] As an optional implementation, the N-gram language model is a concept in computational linguistics and probability theory, referring to a sequence of N words in a given text. When N = 1, it is called unigram; when N = 2, it is called bigram; when N = 3, it is called trigram. Suppose for the text "Dad Du falls into self-blame", when N = 2, through the N-gram language model, the text can be divided into "Dad", "Dad Du", "Du falls", "falls into", "into self", "self-blame", these six character sequences, and M = 6. Suppose the character sequence "Dad" is w1, the character sequence "Dad Du" is w2, and so on, "self-blame" corresponds to w6. In this embodiment, the probability corresponding to each character sequence is the conditional probability relative to the previous character sequence. For example, the probability corresponding to "Dad Du" is P(w2|w1), that is, the probability of each character appearing is only related to the previous character sequence. The probability of the original statement is the product of the probabilities corresponding to all character sequences. In the above original statement "Dad Du falls into self-blame", assuming N = 2, the probability of the original statement is P = P(w1)P(w2|w1)P(w3|w2)P(w4|w3)P(w5|w4)P(w6|w5), and these probabilities can be obtained by counting the number of times the words appear in the corpus. The perplexity of the original statement can be the reciprocal of the probability of the original statement, or the logarithm of the original probability. For example:

[0150]

[0151] Among them, the above P is the probability of the original statement, the above M is the number of words into which the original statement is divided, and PPL is the perplexity of the original statement. In this embodiment, the calculation method of the perplexity of the corrected statement is the same as that of the perplexity of the original statement.

[0152] Optionally, when the decrease ratio of the perplexity of the corrected statement relative to the perplexity of the original statement is greater than a preset threshold, to determine that the original statement is an ungrammatical statement, it includes: determining the difference between the perplexity of the original statement and the perplexity of the corrected statement; determining the decrease ratio to be equal to the ratio of the difference to the perplexity of the original statement; when the decrease ratio is greater than the preset threshold, determining that the original statement is an ungrammatical statement.

[0153] As an optional implementation, assuming the perplexity of the original statement is PPL(A) and the perplexity of the corrected statement is PPL(B), then the decrease ratio is:

[0154]

[0155] Among them, S is the decrease ratio, and PPL(A) - PPL(B) is the difference between the perplexity of the original statement and the perplexity of the corrected statement.

[0156] Optionally, after determining that the original sentence is an incoherent sentence, the method further includes: replacing the original sentence with the revised sentence; or generating corresponding tags for the set of characters to be revised in the original sentence.

[0157] As an optional implementation, after determining that the original sentence is not fluent, the revised sentence can replace the original sentence, or the set of characters to be revised in the original sentence can be marked. Figure 9 As shown in , the original statement is replaced with the modified statement on the user terminal. Figure 10 As shown in the figure, a corresponding mark is generated for the character set to be corrected in the original sentence of the user device. The corresponding mark can also be to mark the characters in the character set to be corrected with special colors such as red and yellow to remind the user that there are incoherent sentences in the published article.

[0158] As an optional implementation, Figure 11 The framework diagram shown includes a judgment module, a generation module, a calculation module, and a confirmation module. The judgment module includes a reduplication module, a reduplication module, a word reduplication module, an intra-sentence repetition module, a sparse judgment module, a missing character module, and a word reversal module. The reduplication module is used to determine whether there are consecutive characters with the same pronunciation in the original sentence, the reduplication module is used to determine whether there are consecutive identical characters in the original sentence, the word reduplication module is used to determine whether there are consecutive identical words in the original sentence, the intra-sentence repetition module is used to determine whether there are characters in the original sentence with a frequency greater than a preset value, the sparse judgment module is used to determine whether there are single characters between consecutive words in the original sentence, the missing character module is used to determine whether there are words in the original sentence with missing characters, and the word reversal module is used to determine whether there are words in the original sentence with reversed order. The generation module is used to process the set of characters to be corrected and generate a corrected sentence from the original sentence. The calculation module is used to calculate whether the reduction ratio of the perplexity of the corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold. The confirmation module is used to determine whether the original sentence is incoherent if the reduction ratio is greater than the preset threshold.

[0159] As an optional implementation, a language model is used to calculate the PPL reduction ratio between the original sentence and the suspected corrected sentence to determine whether the original sentence is incoherent. This method does not require additional annotated data, saving significant annotation manpower. Furthermore, the rule-based module combined with the language model effectively ensures model accuracy. In real-world scenarios, this approach can address incoherent sentences caused by extra or missing words, or incorrect word order, saving significant review time and improving the user experience.

[0160] It is understandable that in the specific implementation of this application, related corpus data such as the corpus of user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0161] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0162] According to another aspect of the embodiment of the present invention, a sentence recognition apparatus for implementing the above sentence recognition method is also provided. Figure 12 As shown, the apparatus includes: a first determining module 1202, configured to determine a set of characters to be corrected in an original sentence, wherein the set of characters to be corrected includes at least one character in the original sentence; a processing module 1204, configured to correct the set of characters to be corrected in the original sentence to obtain a corrected sentence; an acquiring module 1206, configured to acquire the perplexity of the original sentence and the perplexity of the corrected sentence, wherein the perplexity of the original sentence is a parameter obtained by processing the characters in the original sentence using a language model, and the perplexity of the original sentence is used to indicate the degree of incoherence of the original sentence; the perplexity of the corrected sentence is a parameter obtained by processing the characters in the corrected sentence using the language model, and the perplexity of the corrected sentence is used to indicate the degree of incoherence of the corrected sentence; and a second determining module 1208, configured to determine that the original sentence is an incoherent sentence if a decrease ratio of the perplexity of the corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold.

[0163] Optionally, the above-mentioned device is also used to, when the character set to be corrected includes characters to be deleted, delete the characters to be deleted in the original sentence to obtain the corrected sentence; when the character set to be corrected includes characters to be replaced, replace the characters to be replaced with target words in the original sentence to obtain the corrected sentence; when the character set to be corrected includes words to be replaced, replace the words to be replaced with target words in the original sentence to obtain the corrected sentence.

[0164] Optionally, the above-mentioned device is also used to, when the original sentence includes N consecutive identical characters, determine N-1 characters among the N characters as the characters to be deleted, wherein N is a positive integer greater than or equal to 2; delete the N-1 characters from the original sentence; when the original sentence includes M consecutive identical words, determine the characters in M-1 words among the M words as the characters to be deleted, wherein M is a positive integer greater than or equal to 2; delete the M-1 words from the original sentence; when the original sentence includes T consecutive characters with the same pronunciation, determine the characters to be deleted among the T consecutive characters with the same pronunciation, wherein T is a positive integer greater than or equal to 2; and delete the characters to be deleted among the T consecutive characters with the same pronunciation in the original sentence.

[0165] Optionally, the above-mentioned device is also used to delete the kth character from the T consecutive characters with the same pronunciation in the original sentence to obtain the kth corrected sentence, wherein k is greater than or equal to 1 and less than or equal to T; obtain the perplexity of the original sentence and the perplexity of the kth corrected sentence; and determine that the kth character is the character to be deleted when the decrease ratio of the perplexity of the kth corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold.

[0166] Optionally, the above-mentioned device is further used to determine repeated characters in the original sentence, wherein the number of the repeated characters in the original sentence is greater than a preset value; and determine the characters to be deleted from the repeated characters in the original sentence.

[0167] Optionally, the above-mentioned device is also used to delete the i-th repeated character in the original sentence to obtain the i-th revised sentence, where i is greater than or equal to 1; obtain the perplexity of the original sentence and the perplexity of the i-th revised sentence; when the decrease ratio of the perplexity of the i-th revised sentence relative to the perplexity of the original sentence is greater than the preset threshold, determine that the i-th repeated character is the character to be deleted.

[0168] Optionally, the above-mentioned device is also used to delete the i+1th repeated character in the original sentence to obtain the i+1th revised sentence when the ratio of decrease in the perplexity of the i-th revised sentence relative to the perplexity of the original sentence is less than or equal to the preset threshold; obtain the perplexity of the original sentence and the perplexity of the i+1th revised sentence; and determine that the i+1th repeated character is the character to be deleted when the ratio of decrease in the perplexity of the i+1th revised sentence relative to the perplexity of the original sentence is greater than a preset threshold.

[0169] Optionally, the above-mentioned device is also used to, when the original sentence includes a first uncommon word, delete the j-th character in the first uncommon word in the original sentence to obtain the j-th corrected sentence, wherein j is greater than or equal to 1, and the number of times the first uncommon word appears in the corpus is less than a preset value; obtain the perplexity of the original sentence and the perplexity of the j-th corrected sentence; when the decrease ratio of the perplexity of the j-th corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold, determine that the j-th character is the character to be deleted.

[0170] Optionally, the above-mentioned device is also used to delete the j+1th character in the uncommon word in the original sentence to obtain the j+1th revised sentence when the ratio of decrease in the perplexity of the j-th revised sentence relative to the perplexity of the original sentence is less than or equal to a preset threshold; obtain the perplexity of the original sentence and the perplexity of the j+1th revised sentence; and determine that the j+1th repeated character is the character to be deleted when the ratio of decrease in the perplexity of the j+1th revised sentence relative to the perplexity of the original sentence is greater than a preset threshold.

[0171] Optionally, the apparatus is further configured to, when the original sentence includes a single character, determine the single character as the character to be replaced, wherein the single character is a character located between two adjacent words in the original sentence.

[0172] Optionally, the above-mentioned device is also used to search for words in the corpus with the character to be replaced as the first or last character; among the words found, the word with the highest number of occurrences in the corpus is determined as the target word; and the character to be replaced is replaced with the target word in the original sentence.

[0173] Optionally, the above-mentioned device is also used to determine the second uncommon word as the word to be replaced when the original sentence includes the second uncommon word, wherein the number of times the second uncommon word appears in the corpus is less than a preset value, and the target word is a word obtained by sorting the characters in the second uncommon word in reverse order, and the number of times the target word appears in the corpus is greater than or equal to the preset value.

[0174] Optionally, the above-mentioned device is also used to input the original sentence into the N-gram language model, divide the original sentence into multiple character sequences through the N-gram language model, wherein each character sequence includes N adjacent characters in the original sentence, wherein N is a positive integer greater than or equal to 2; determine the conditional probability of each character sequence in the multiple character sequences relative to the previous character sequence as the probability corresponding to each character sequence; determine the probability corresponding to the original sentence through the probability corresponding to each character sequence; and determine the perplexity of the original sentence based on the probability corresponding to the original sentence.

[0175] Optionally, the above-mentioned device is also used to determine the difference between the perplexity of the original sentence and the perplexity of the revised sentence; determine the reduction ratio to be equal to the ratio of the difference to the perplexity of the original sentence; and when the reduction ratio is greater than the preset threshold, determine that the original sentence is an incoherent sentence.

[0176] Optionally, the apparatus is further configured to, after determining that the original sentence is an incoherent sentence, replace the original sentence with the revised sentence; and generate corresponding tags for the set of characters to be revised in the original sentence.

[0177] According to another aspect of the embodiment of the present invention, an electronic device for implementing the above sentence recognition method is also provided. The electronic device may be Figure 1 The terminal device or server shown in FIG. This embodiment is described by taking the electronic device as a server as an example. Figure 13 As shown, the electronic device includes a memory 1302 and a processor 1304. The memory 1302 stores a computer program, and the processor 1304 is configured to execute the steps in any of the above method embodiments through the computer program.

[0178] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0179] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0180] S1, determining a character set to be corrected in an original sentence, wherein the character set to be corrected includes at least one character in the original sentence;

[0181] S2, performing correction processing on the set of characters to be corrected in the original sentence to obtain a corrected sentence;

[0182] S3, obtaining the perplexity of the original sentence and the perplexity of the revised sentence, wherein the perplexity of the original sentence is a parameter obtained by processing characters in the original sentence using a language model, and the perplexity of the original sentence is used to indicate the degree of incoherence of the original sentence; the perplexity of the revised sentence is a parameter obtained by processing characters in the revised sentence using the language model, and the perplexity of the revised sentence is used to indicate the degree of incoherence of the revised sentence;

[0183] S4: When the decrease ratio of the perplexity of the modified sentence relative to the perplexity of the original sentence is greater than a preset threshold, determine that the original sentence is an incoherent sentence.

[0184] Alternatively, those skilled in the art will appreciate that Figure 13 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 13 It does not limit the structure of the electronic device. For example, the electronic device may also include Figure 13 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 13 Different configurations shown.

[0185] Among them, the memory 1302 can be used to store software programs and modules, such as the program instructions / modules corresponding to the sentence recognition method and device in the embodiment of the present invention. The processor 1304 executes various functional applications and data processing by running the software programs and modules stored in the memory 1302, that is, realizing the above-mentioned sentence recognition method. The memory 1302 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1302 may further include a memory remotely located relative to the processor 1304, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1302 can be used specifically, but not limited to, to store information such as sample characteristics of items and target virtual resource accounts. As an example, such as Figure 13 As shown, the memory 1302 may include, but is not limited to, the first determination module 1202, the processing module 1204, the acquisition module 1206, and the second determination module 1208 in the sentence recognition device. In addition, it may also include, but is not limited to, other module units in the sentence recognition device, which will not be repeated in this example.

[0186] Optionally, the transmission device 1306 is configured to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one embodiment, the transmission device 1306 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 1306 is a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0187] In addition, the electronic device further includes: a display 1308 for displaying the original sentence and the revised sentence; and a connection bus 1310 for connecting various module components in the electronic device.

[0188] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting multiple nodes through network communication. The nodes may form a peer-to-peer (P2P) network, and any computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0189] According to one aspect of the present application, a computer program product is provided, comprising a computer program / instructions containing program code for executing the method illustrated in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion 1409 and / or installed from a removable medium 1411. When the computer program is executed by the central processing unit 1401, the various functions provided in the embodiments of the present application are performed.

[0190] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0191] Figure 14 The block diagram schematically shows a computer system structure of an electronic device used to implement an embodiment of the present application.

[0192] It should be noted that Figure 14 The computer system 1400 of the electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0193] like Figure 14As shown, the computer system 1400 includes a central processing unit 1401 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1402 (ROM) or the program loaded from the storage part 1408 into the random access memory 1403 (RAM). Various programs and data required for system operation are also stored in the random access memory 1403. The central processing unit 1401, the read-only memory 1402 and the random access memory 1403 are connected to each other via a bus 1404. An input / output interface 1405 (i.e., an I / O interface) is also connected to the bus 1404.

[0194] The following components are connected to the input / output interface 1405: an input section 1406 including a keyboard, a mouse, and the like; an output section 1407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1408 including a hard disk; and a communication section 1409 including a network interface card such as a local area network card or a modem. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the input / output interface 1405 as needed. Removable media 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1410 as needed, so that computer programs read therefrom can be installed into the storage section 1408 as needed.

[0195] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1409 and / or installed from a removable medium 1411. When the computer program is executed by the central processing unit 1401, the various functions defined in the system of the present application are performed.

[0196] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementations described above.

[0197] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0198] S1, determining a character set to be corrected in an original sentence, wherein the character set to be corrected includes at least one character in the original sentence;

[0199] S2, performing correction processing on the set of characters to be corrected in the original sentence to obtain a corrected sentence;

[0200] S3, obtaining the perplexity of the original sentence and the perplexity of the revised sentence, wherein the perplexity of the original sentence is a parameter obtained by processing characters in the original sentence using a language model, and the perplexity of the original sentence is used to indicate the degree of incoherence of the original sentence; the perplexity of the revised sentence is a parameter obtained by processing characters in the revised sentence using the language model, and the perplexity of the revised sentence is used to indicate the degree of incoherence of the revised sentence;

[0201] S4: When the decrease ratio of the perplexity of the modified sentence relative to the perplexity of the original sentence is greater than a preset threshold, determine that the original sentence is an incoherent sentence.

[0202] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0203] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing one or more computer devices (such as personal computers, servers, or network devices) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0204] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0206] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0207] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0208] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A sentence recognition method, characterized in that: include: In the case that the original sentence includes an object to be replaced, the object to be replaced is added to the set of objects to be corrected, wherein the object to be replaced includes: a word to be replaced and a character to be replaced; When the original sentence includes multiple candidate characters to be deleted, traversing each of the multiple candidate characters to be deleted in turn, and performing the following steps: correcting the current candidate character to be deleted in the original sentence to obtain a candidate corrected sentence, and when the perplexity of the candidate corrected sentence decreases by a ratio greater than a preset threshold relative to the perplexity of the original sentence, determining the current candidate character to be deleted as a character to be deleted, and adding the current candidate character to the set of objects to be corrected, wherein the perplexity is used to indicate the degree of incoherence of the sentence; Performing correction processing on the set of objects to be corrected in the original sentence to obtain a corrected sentence; When a decrease ratio of the perplexity of the modified sentence relative to the perplexity of the original sentence is greater than a preset threshold, the original sentence is determined to be an incoherent sentence.

2. The method according to claim 1, characterized in that The step of performing correction processing on the set of objects to be corrected in the original statement to obtain a corrected statement includes one of the following: In a case where the set of objects to be revised includes characters to be deleted, deleting the characters to be deleted from the original sentence to obtain the revised sentence; In a case where the set of objects to be corrected includes characters to be replaced, replacing the characters to be replaced with target words in the original sentence to obtain the corrected sentence; In the case that the set of objects to be corrected includes words to be replaced, the words to be replaced are replaced with target words in the original sentence to obtain the corrected sentence.

3. The method according to claim 2, wherein Before performing correction processing on the set of objects to be corrected in the original sentence to obtain a corrected sentence, the method further includes: In the case where the original sentence includes N consecutive identical characters, adding N-1 characters of the N characters to the set of objects to be corrected, where N is a positive integer greater than or equal to 2; or In the case that the original sentence includes M consecutive identical words, characters in M-1 words among the M words are added to the set of objects to be corrected, where M is a positive integer greater than or equal to 2.

4. The method according to claim 3, characterized in that The correcting the current candidate character to be deleted in the original sentence to obtain a candidate corrected sentence includes: deleting a k-th character from T consecutive characters with the same pronunciation in the original sentence to obtain a k-th corrected sentence, wherein k is greater than or equal to 1 and less than or equal to T, the k-th character is used to represent the current candidate character to be deleted, and the k-th corrected sentence is used to represent the candidate corrected sentence; The method comprises: obtaining the perplexity of the original sentence and the perplexity of the k-th revised sentence; and determining the k-th character as the character to be deleted and adding the k-th character to the set of objects to be revised when the perplexity of the candidate revised sentence decreases at a ratio greater than a preset threshold relative to the perplexity of the original sentence. The method comprises: obtaining the perplexity of the original sentence and the perplexity of the k-th revised sentence; and determining the k-th character as the character to be deleted and adding the k-th character to the set of objects to be revised when the perplexity of the k-th revised sentence decreases at a ratio greater than a preset threshold relative to the perplexity of the original sentence.

5. The method according to claim 2, characterized in that The method further comprises: In a case where the candidate character to be deleted is a repeated character, the correcting the current candidate character to be deleted in the original sentence to obtain a candidate corrected sentence includes: deleting the i-th repeated character in the original sentence to obtain the i-th corrected sentence, wherein i is greater than or equal to 1, wherein the number of repeated characters in the original sentence is greater than a preset value, the i-th repeated character is used to represent the current candidate character to be deleted, and the i-th corrected sentence is used to represent the candidate corrected sentence; In the case that the candidate character to be deleted is a repeated character, and when the ratio of decrease in the perplexity of the candidate revised sentence relative to the perplexity of the original sentence is greater than a preset threshold, the current candidate character to be deleted is determined as the character to be deleted, and the current candidate character to be deleted is added to the set of objects to be corrected, including: obtaining the perplexity of the original sentence and the perplexity of the i-th revised sentence; and when the ratio of decrease in the perplexity of the i-th revised sentence relative to the perplexity of the original sentence is greater than the preset threshold, determining the i-th repeated character as the character to be deleted, and adding the i-th repeated character to the set of objects to be corrected.

6. The method according to claim 2, characterized in that The method further comprises: In a case where the candidate character to be deleted is a character in an uncommon word, the correcting the current candidate character to be deleted in the original sentence to obtain a candidate corrected sentence comprises: in a case where the original sentence includes a first uncommon word, deleting the jth character in the first uncommon word in the original sentence to obtain a jth corrected sentence, wherein j is greater than or equal to 1, and the number of occurrences of the first uncommon word in the corpus is less than a preset value, wherein the jth character is used to represent the current candidate character to be deleted, and the jth corrected sentence is used to represent the candidate corrected sentence; In the case that the candidate character to be deleted is a character in an uncommon word, and when the ratio of decrease in the perplexity of the candidate revised sentence relative to the perplexity of the original sentence is greater than a preset threshold, the current candidate character to be deleted is determined as the character to be deleted, and the current candidate character to be deleted is added to the set of objects to be corrected, including: obtaining the perplexity of the original sentence and the perplexity of the j-th revised sentence; and when the ratio of decrease in the perplexity of the j-th revised sentence relative to the perplexity of the original sentence is greater than a preset threshold, determining the j-th character as the character to be deleted, and adding the character to be deleted to the set of objects to be corrected.

7. The method according to claim 2, characterized in that Before performing correction processing on the set of objects to be corrected in the original sentence to obtain a corrected sentence, the method further includes: In a case where the original sentence includes a single character, determining the single character as the character to be replaced, wherein the single character is a character located between two adjacent words in the original sentence; The single character is added to the set of objects to be corrected.

8. The method according to claim 7, characterized in that The method further comprises: searching a corpus for a word with the character to be replaced as the first character or the last character; determining the word with the highest number of occurrences in the corpus as the target word among the words found; Replacing the characters to be replaced with target characters in the original sentence to obtain the revised sentence includes: replacing the characters to be replaced with the target word in the original sentence.

9. The method according to claim 2, characterized in that The method further comprises: In a case where the original sentence includes a second uncommon word, the second uncommon word is determined as the word to be replaced, wherein the number of times the second uncommon word appears in the corpus is less than a preset value, and the target word is a word obtained by sorting the characters in the second uncommon word in reverse order, and the number of times the target word appears in the corpus is greater than or equal to the preset value.

10. The method according to any one of claims 1 to 9, characterized in that Obtaining the perplexity of the original sentence includes: Inputting the original sentence into an N-gram language model, dividing the original sentence into a plurality of character sequences using the N-gram language model, wherein each character sequence includes N adjacent characters in the original sentence, wherein N is a positive integer greater than or equal to 2; Determining the conditional probability of each character sequence in the plurality of character sequences relative to the previous character sequence as the probability corresponding to each character sequence; Determining the probability corresponding to the original sentence based on the probability corresponding to each character sequence; The perplexity of the original sentence is determined according to the probability corresponding to the original sentence.

11. The method according to any one of claims 1 to 9, characterized in that When a decrease ratio of the perplexity of the modified sentence relative to the perplexity of the original sentence is greater than a preset threshold, determining that the original sentence is an incoherent sentence includes: Determining a difference between the perplexity of the original sentence and the perplexity of the modified sentence; Determining the reduction ratio to be equal to the ratio of the difference to the perplexity of the original sentence; When the decrease ratio is greater than the preset threshold, the original sentence is determined to be an incoherent sentence.

12. The method according to any one of claims 1 to 9, characterized in that After determining that the original sentence is an incoherent sentence, the method further includes: replacing the original statement with the revised statement; or Generate corresponding tags for the set of objects to be modified in the original sentence.

13. A sentence recognition device, characterized in that: include: A first determination module is configured to, when an original sentence includes an object to be replaced, add the object to be replaced to a set of objects to be corrected, wherein the object to be replaced includes: a word to be replaced and a character to be replaced; and when the original sentence includes multiple candidate characters to be deleted, sequentially traverse each candidate character to be deleted in the multiple candidate characters to be deleted, and perform the following steps: correcting a current candidate character to be deleted in the original sentence to obtain a candidate corrected sentence; and when a decrease ratio of the perplexity of the candidate corrected sentence relative to the perplexity of the original sentence is greater than a preset threshold, determining the current candidate character to be deleted as a character to be deleted, and adding the current candidate character to be deleted to the set of objects to be corrected, wherein the perplexity is used to indicate the degree of incoherence of the sentence; a processing module, configured to perform correction processing on the set of objects to be corrected in the original sentence to obtain a corrected sentence; The second determination module is configured to determine that the original sentence is an incoherent sentence when a decrease ratio of the perplexity of the revised sentence relative to the perplexity of the original sentence is greater than a preset threshold.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein the method according to any one of claims 1 to 12 is executed when the program is executed.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

16. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 12 through the computer program.

Images