Sentence boundary recognition method, sentence boundary recognition apparatus, and electronic device

Abstract

The application provides a sentence boundary recognition method, a sentence boundary recognition device and an electronic equipment. The method comprises the following steps: obtaining a text to be recognized; inputting a string corresponding to the text to be recognized into a finite state converter for splitting to obtain at least one slice, the slice being used for indicating a character segment in the string and a slice type corresponding to the character segment, and a state in the finite state converter being constructed based on a character type; determining a target slice in the at least one slice based on a preset sentence breaking rule, the target slice being used for indicating a sentence boundary of the text to be recognized. The application provides the sentence boundary recognition method, the sentence boundary recognition device and the electronic equipment, which can quickly determine the sentence boundary of the text to be recognized under the premise of as few loop traversals of the string as possible, thereby improving the processing efficiency of text sentence breaking.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a sentence boundary recognition method, a sentence boundary recognition device, and an electronic device. Background Technology

[0002] In existing technologies, rule-based sentence segmentation systems require that the input segment be standard input, and that the document input must be error-free and conform to the standard in order to ensure the accuracy of sentence segmentation.

[0003] In standard input documents, there are many ambiguous characters that make sentence segmentation difficult. For example, the character "." in English sentences can be used not only as the end of a sentence but also as a decimal point; or a single quotation mark can be used not only to indicate a quotation but also to indicate a possessive noun.

[0004] When dealing with these ambiguous parts, it is often necessary to rely on inefficient string processing methods such as regular expressions, which is very time-consuming. Summary of the Invention

[0005] This invention provides a sentence boundary recognition method, a sentence boundary recognition device, and an electronic device to solve the problem of low efficiency in text segmentation in the prior art.

[0006] This invention provides a method for identifying statement boundaries, comprising:

[0007] Obtain the text to be recognized;

[0008] The string corresponding to the text to be identified is input into a finite state converter for splitting to obtain at least one slice. The slice is used to indicate the character fragments in the string and the slice type corresponding to the character fragments. The state in the finite state converter is constructed based on the character type.

[0009] Based on preset sentence segmentation rules, a target slice is determined in the at least one slice, and the target slice is used to indicate the sentence boundaries of the text to be identified.

[0010] In some embodiments, the character type includes at least one of the following:

[0011] Ordinary characters, including characters that do not affect clause segmentation;

[0012] Clause-segmenting characters, including ordinary clause-segmenting characters and special clause-segmenting characters. Ordinary clause-segmenting characters include characters that only have clause-segmenting function, while special clause-segmenting characters include all characters with clause-segmenting function except for the ordinary clause-segmenting characters.

[0013] Interval symbols, the interval symbols including at least one type of brackets, wherein an open interval symbol in the interval symbol is used to indicate the left half of the interval symbol, and a closed interval symbol in the interval symbol is used to indicate the right half of the interval symbol;

[0014] Space character;

[0015] Quotation characters;

[0016] The slice type includes at least one of the following: ordinary character type, ordinary clause character type, special clause character type, open interval symbol type, closed interval symbol type, space character type, and quotation mark character type.

[0017] In some embodiments, the preset sentence segmentation rules include at least one of the following:

[0018] If the interval symbol is not present in at least one slice, the slice of type ordinary clause character is determined as the target slice;

[0019] If the interval symbol is not present in at least one slice, determine the next slice of the slice with the special clause character type;

[0020] If the first letter of the latter slice is a capital letter, the slice of the special clause character type is determined as the target slice.

[0021] In some embodiments, the preset sentence segmentation rules include at least one of the following:

[0022] If the interval symbol is present in at least one slice, the slice whose type is the ordinary clause character type or the special clause character type is determined as a non-target slice;

[0023] If the interval symbol is not present in at least one slice, determine the slice preceding the slice of the special clause character type;

[0024] If the characters in the previous slice meet the preset conditions, the slice of the special clause character type is determined as a non-target slice;

[0025] The preset conditions include at least one of the following:

[0026] The character fragment of the previous slice includes an uppercase letter and at least one lowercase letter;

[0027] The character fragment of the previous slice includes at most two numbers;

[0028] The character fragments of the previous slice consist of a single letter.

[0029] In some embodiments, inputting the string corresponding to the text to be identified into a finite-state converter for splitting to obtain at least one slice includes:

[0030] Upon obtaining the first slice of the open interval symbol type, push the first slice onto the target stack;

[0031] In the case of obtaining a second slice of the closed interval symbol type, the second slice is paired with the top element of the target stack;

[0032] If it is determined that the first slice is successfully paired with the top element of the stack, a pop operation is performed on the top element of the stack.

[0033] In some embodiments, after obtaining a first slice of the open interval symbol type and pushing the first slice onto the target stack, the method further includes:

[0034] If a third slice of the quoted character type is obtained, the third slice is paired with the top element of the target stack;

[0035] If it is determined that the third slice is successfully paired with the top element of the stack, a pop operation is performed on the top element of the stack.

[0036] If it is determined that the third slice cannot be paired with the top element of the stack, the third slice is pushed onto the target stack.

[0037] The present invention also provides a statement boundary recognition device, comprising:

[0038] The acquisition module is used to acquire the text to be recognized;

[0039] The splitting module is used to input the string corresponding to the text to be identified into a finite state converter for splitting to obtain at least one slice. The slice is used to indicate the character fragments in the string and the slice type corresponding to the character fragments. The state in the finite state converter is constructed based on the character type.

[0040] The determination module is used to determine a target slice in the at least one slice based on a preset sentence segmentation rule, wherein the target slice is used to indicate the sentence boundaries of the text to be identified.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the statement boundary recognition method as described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the statement boundary recognition method as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the statement boundary recognition method as described above.

[0044] The sentence boundary recognition method, sentence boundary recognition device, and electronic device provided by this invention, through a finite state converter, can quickly split the file to be recognized and obtain slices with minimal loop traversal of the string, thus improving recognition efficiency; combined with preset sentence segmentation rules, the target slice can be accurately determined, which not only saves time but also ensures recognition accuracy. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the statement boundary recognition method provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of the statement boundary recognition device provided by the present invention;

[0048] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] The execution subject of the statement boundary recognition method provided by this invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This invention does not impose specific limitations.

[0051] The following example, using a computer executing the statement boundary recognition method provided by this invention, illustrates the technical solution of this invention in detail.

[0052] Figure 1 This is a flowchart illustrating the statement boundary recognition method provided by the present invention. (Refer to...) Figure 1 The statement boundary recognition method provided by the present invention includes steps 110, 120 and 130.

[0053] Step 110: Obtain the text to be recognized;

[0054] Step 120: Input the string corresponding to the text to be recognized into the finite state converter for splitting to obtain at least one slice. The slice is used to indicate the character fragments in the string and the slice type corresponding to the character fragments. The state in the finite state converter is constructed based on the character type.

[0055] Step 130: Based on preset sentence segmentation rules, determine the target slice in at least one slice. The target slice is used to indicate the sentence boundaries of the text to be identified.

[0056] In step 110, the text to be identified can be text that requires sentence segmentation recognition, that is, text that requires detection of sentence breaks. This embodiment does not specifically limit the language of the text to be identified; for example, it can be English text.

[0057] In step 120, the string corresponding to the text to be recognized is traversed through a finite state converter to split the string into slices.

[0058] A splitting operation can be an operation that extracts a substring (a portion of a string) from a string, generating at least one slice. At least one slice can be stored in a slice list, which is used to sequentially store the slice's value and type.

[0059] In practice, a slice is defined using a pair of square brackets, a character fragment, and a slice type, i.e., [character fragment slice type]. This slice is similar to word segmentation, but the slice is formed based on the purpose of sentence segmentation.

[0060] There is a one-to-one correspondence between slice types and character types. The character type for a single character can use a character set table provided by an international standards organization, such as UTF-8.

[0061] In this embodiment, characters and slices are first classified as follows, and the character type includes at least one of the following:

[0062] I. Ordinary characters, which include characters that do not affect clause segmentation. For example, letters, numbers, or symbols such as "—" that do not affect clause segmentation.

[0063] II. Clause characters. Clause characters include ordinary clause characters and special clause characters. Ordinary clause characters include characters that only have clause-breaking functions, while special clause characters include all characters with clause-breaking functions except for ordinary clause characters.

[0064] Third, clause-breaking characters are characters that have the function of breaking sentences. Based on the function of clause-breaking characters, they can be further divided into ordinary clause-breaking characters and special clause-breaking characters.

[0065] Ordinary clause separators are unambiguous symbols that can directly separate clauses, such as: colon ":", comma ",", question mark "?", exclamation mark "!", semicolon ";", ellipsis "..." or "...", etc. Ordinary clause separators can serve as unambiguous sentence breaks within a string.

[0066] Special clause-breaking characters are those that can be used as clause-breaking symbols in certain situations but not in others. For example, the character "." in English sentences can be used not only at the end of a sentence but also as a decimal point. Of course, such ambiguous characters also exist in other types of sentences, but these will not be specifically limited here.

[0067] IV. Interval symbols, which include at least one type of brackets. Open interval symbols are used to indicate the left half of the interval symbol, and closed interval symbols are used to indicate the right half of the interval symbol.

[0068] The interval symbol can be parentheses of type “()”, “{}”, “[]”, or “<>”.

[0069] The open interval symbol is the left half of the interval symbol such as "(", "{", "[" or "<".

[0070] The closed interval symbol is the right half of the interval symbol such as “)”, “}”, “]” or ">”.

[0071] 5. Space character, which is the character that represents a space.

[0072] 6. Quotation characters, including single quotes and double quotes.

[0073] In actual execution, the character type corresponding to a regular character can be represented by "nn";

[0074] The character type corresponding to the space character can be set to "sp";

[0075] The character type corresponding to a regular clause character can be represented by "ns";

[0076] The character type corresponding to special clause characters can be represented by "fs";

[0077] The character type corresponding to the open interval symbol can be represented by "os";

[0078] The slice type corresponding to the closed interval symbol can be represented by "cs";

[0079] The slice type corresponding to the quotation marks can be represented by "qs".

[0080] Accordingly, the slice type may include at least one of the following: ordinary character type, ordinary clause character type, special clause character type, open interval symbol type, closed interval symbol type, space character type, and quotation mark character type.

[0081] In this embodiment, based on the character type, the slice type can also be represented as at least one of the following: ordinary character type nn; space character type sp; ordinary clause character type ns; special clause character type fs; open interval symbol type os; closed interval symbol type cs; quotation mark character type qs.

[0082] For example, LiLei said: “I am OK!” can be divided into [LiLei nn], [said:nn], [“qs], [I nn], [am nn], [OK nn], [!ns] and [“qs].

[0083] It should be noted that finite state transducers (FSTs) are currently widely used in speech recognition, natural language search, and processing.

[0084] A finite automaton (FA) consists of a finite set of states and state transitions, each transition having at least one label. The most basic FA is the finite state acceptor (FSA). For a given input sequence, the FSA returns either "accept" or "do not accept".

[0085] FST is an extension of FSA, and it has an output label for each state transition, called an input-output label pair. Using these label pairs, FST can describe a set of rule transitions or a transformation from one set of symbol sequences to another. For example, FST can transform the symbol sequence "a, b, c, d" into another symbol sequence "z, y, x, w".

[0086] The state transitions of a finite state converter are based on changes in character type. Therefore, the states of a finite state converter can include: initialization state (init), nn state, sp state, ns state, fs state, os state, cs state, qs state, etc.

[0087] The state transition process of the finite state converter in this embodiment is as follows:

[0088] Initialize state init:

[0089] 1. The current input character is of type nn. The current character is added to the current slice, the current slice type is set to nn, and the state transitions to nn.

[0090] 2. The current input character type is sp. The current character is discarded, the status changes to sp, and a space precedes the paragraph.

[0091] 3. The current input character is of type ns. The current character is added to the current slice, and the state changes to nn. This is generally an incorrect input and should be treated as a normal character.

[0092] 4. The current input character type is fs. The current character is added to the current slice, and the state transitions to nn. This is generally an incorrect input and should be treated as a normal character.

[0093] 5. The current input character is of type os. The current character is added to the slice list as an independent slice, the slice type is set to os, and the state transitions to os.

[0094] 6. The current input character type is cs. The current character is added to the current slice, and the state transitions to nn. This case is treated as a normal character.

[0095] 7. The current input character type is qs. The current character is added to the slice list as an independent slice, the slice type is set to qs, and the state transitions to qs.

[0096] nn state:

[0097] 1. The current input character type is nn. The current character is added to the current slice, and the state changes to nn.

[0098] 2. The current input character type is sp. The current character is discarded, the current slice is added to the slice list, and the state changes to sp.

[0099] 3. The current input character type is ns. The current slice is added to the slice list, and the previous character is added to the slice list as a separate slice, with its slice type set to ns. The state transitions to ns. Unlike fs, the statement "hello!world" will still be segmented into clauses. Note that the statement "hello!world" does not include the two quotation marks.

[0100] 4. The current input character type is fs. The current character is added to the current slice, and the state transitions to fs. For example, the statement "ab" cannot be segmented into a clause. Note that the statement "ab" does not include the two quotation marks.

[0101] 5. The current input character type is os. The current character is added to the current slice, and the state transitions to nn. Inputs of the form "are{" are generally considered incorrect and are treated as ordinary characters. Note that the statement "are{" does not include the first and last quotation marks.

[0102] 6. The current input character type is cs. The current character is added to the current slice, and the state is switched to cs.

[0103] 7. The current input character type is qs. The current character is added to the current slice. The state transitions to qs.

[0104] sp state:

[0105] 1. The current input character is of type nn. The current character is added to the current slice, the current slice type is set to nn, and the state transitions to nn.

[0106] 2. The current input character type is sp. The current character is discarded, the state changes to sp, and extra spaces are added to the segment.

[0107] 3. The current input character is of type ns. The current character is added to the current slice, and the state changes to nn. This is generally an incorrect input and should be treated as a normal character.

[0108] 4. The current input character type is fs. The current character is added to the current slice, and the state transitions to nn. This is generally an incorrect input and should be treated as a normal character.

[0109] 5. The current input character type is os. The current character is added to the slice list as a separate slice, and the slice type is set to os. The state transitions to os.

[0110] 6. The current input character type is cs. The current character is added to the current slice, and the state transitions to nn. This case is treated as a normal character, for example, the statement "are]you". Here, "are]you" does not include the two quotation marks.

[0111] 7. The current input character type is qs. The current character is added to the slice list as an independent slice, the slice type is set to qs, and the state transitions to qs.

[0112] ns state:

[0113] 1. The current input character type is nn. Pop the last slice from the slice list and add it to the current slice. Add the current character to the current slice and set the current slice type to nn. Transition to nn. An error like this occurs because the statement `.hello` treats the previous ns character as a normal character.

[0114] 2. The current input character type is sp. The current character is discarded, the current slice is added to the slice list (if the slice is empty, nothing is done), and the state changes to sp.

[0115] 3. The current input character type is ns. Change the type of the last slice in the slice list to nn, add the current character as an independent slice to the slice list, set the slice type to ns, and switch the state to ns. For characters like multiple consecutive symbols "!?", the last one is a period, and the others are treated as ordinary characters. Note that the symbol "!?" does not include the two quotation marks.

[0116] 4. The current input character type is fs. Change the type of the last slice in the slice list to nn, add the current character as an independent slice to the slice list, set the slice type to fs, and switch the state to fs. For characters like multiple consecutive symbols "?.", the last one is a period, and the others are treated as ordinary characters. Note that the symbol "?." does not include the two quotation marks.

[0117] 5. The current input character type is os. The current character is added to the current slice, and the slice type is set to nn. The state transitions to nn. Error symbols like "?(" are treated as ordinary characters. Note that the "?(" symbol does not include the two quotation marks.)

[0118] 6. The current input character type is cs. The current character is added to the slice list as a separate slice, and the slice type is set to cs. The state transitions to cs. For example: two consecutive symbols "?)". Note that the symbols "?)" do not include the two quotation marks.

[0119] 7. The current input character type is qs. The current character is added to the slice list as a separate slice, and the slice type is set to qs. The state transitions to qs. For example: two consecutive symbols "?". Note that the symbol "?" does not include the two double quotes.

[0120] fs state:

[0121] 1. The current input character is of type nn. The current character is added to the current slice, the current slice type is set to nn, and the state transitions to nn.

[0122] 2. The current input character type is sp. The current character is discarded, the last character of the last slice in the slice list is separated into a separate slice and added to the slice list, and the slice type is set to ns. The state is changed to sp.

[0123] 3. The current input character type is ns. The current character is added to the slice list as an independent slice, and the slice type is set to ns, and the state transitions to ns. This is similar to multiple consecutive clause markers ".". Note that the "." symbol does not include the two quotation marks.

[0124] 4. The current input character type is fs. The current character is added to the last slice in the slice list, and the state transitions to fs. It takes the form of multiple consecutive symbols "..)". The symbols "..)" do not include the two quotation marks.

[0125] 5. The current input character type is os. The current character is added to the last slice in the slice list, and the state transitions to nn. The symbol ".(" is treated as a normal character. Note that the symbol ".(" does not include the two quotation marks.

[0126] 6. The current input character type is cs. The last character of the last slice in the slice list is separated into a separate slice and added to the slice list, with its slice type set to fs. The current character is added to the slice list as a separate slice, with its slice type set to cs. The state transitions to cs. The forms are ".)" and ".>", where ".)" and ".>" do not include the two quotation marks.

[0127] 7. The current input character type is qs. The last character of the last slice in the slice list is added to the slice list as a separate slice, and its slice type is set to fs. The current character is added to the slice list as a separate slice, and its slice type is set to qs. The state transitions to qs. The characters are in the form of the symbols "." and ".". The symbol "." does not include the first and last quotation marks, and the symbol "." does not include the first and last quotation marks.

[0128] OS status:

[0129] 1. The current input character type is nn. The current character is added to the current slice, the current slice type is set to nn, and the state transitions to nn. For example, for a character like "<a", the character "<a" does not include the two quotation marks.

[0130] 2. The current input character type is sp. The current character is discarded, the type of the last slice in the slice list is modified to nn, and the state transitions to sp. (If there is a space after an opening symbol, it is not considered an opening symbol to prevent confusion such as a<5, b>10).

[0131] 3. The current input character type is ns. The current character is added to the current slice, and the current slice type is set to nn. The state transitions to nn. For example, for a symbol like "<?", a similar situation may occur and it is treated as an ordinary character. The symbol "<?" does not include the two quotation marks.

[0132] 4. The current input character type is fs. The current character is added to the current slice, the current slice type is set to nn, and the state transitions to nn. For example, for symbols like "<." and "(.", they may be input errors and are treated as ordinary characters. The above symbols "<." and "(." do not include the two quotation marks respectively.

[0133] 5. The current input character type is os. The current character is added as an independent slice to the slice list, the slice type is set to os, the state transitions to os, and interval nesting occurs.

[0134] 6. The current input character type is cs. The current character is added as an independent slice to the slice list, the slice type is set to cs, and the state transitions to cs. For example, for a symbol like "()<)", the above symbol "()<)" does not include the two quotation marks.

[0135] 7. The current input character type is qs. The current character is added as an independent slice to the slice list, the slice type is set to qs, and the state transitions to qs. For example, for a symbol like "(‘<“”, the symbol "(‘<“” does not include the first and the last quotation marks.

[0136] cs state:

[0137] 1. The current input character type is nn. The type of the last slice in the slice list is modified to nn, the current character is added to the current slice, the current slice type is set to nn, and the state transitions to nn. For example, a character like ")a" is an incorrect input and is not considered a closing interval symbol. The character ")a" does not include the two quotation marks.

[0138] 2. The current input character type is sp. The current character is discarded, the last character of the last slice in the slice list is separated and added as a slice to the slice list, and the slice type is set to cs, and the state transitions to sp.

[0139] 3. The current input character is of type ns. The current character is added to the slice list as a separate slice, and the slice type is set to ns. The state transitions to ns. The form is similar to the symbol ")?", where ")?" does not include the two quotation marks.

[0140] 4. The current input character type is fs. The current character is added to the slice list as an independent slice, and the slice type is set to fs. The state transitions to fs. The input character is in the form of the symbol ".", where "." does not include the two quotation marks.

[0141] 5. The current input character type is os. The current character is added to the current slice, the current slice type is set to nn, and the state transitions to nn. This may be erroneous input; treat it as a normal character. Symbols like “)<)()}{”, where “)<)()}{” are not enclosed in quotation marks.

[0142] 6. The current input character type is cs. The current character is added to the slice list as an independent slice, and the slice type is set to cs. The state transitions to cs. For example, nested intervals ")))>", where ")))>" does not include the two quotation marks.

[0143] 7. The current input character type is qs. The current character is added to the slice list as an independent slice, and the slice type is set to qs. The state transitions to qs. For example, nested intervals ")'" and ">", where ")'" does not include the first and last quotation marks, and ">" does not include the first and last quotation marks.

[0144] qs status:

[0145] 1. The current input character type is nn. The current character is added to the current slice, the current slice type is set to nn, and the state transitions to nn. For example: the characters "a" and "'a", where "a" does not include the first and last quotation marks, and "'a" does not include the first and last quotation marks.

[0146] 2. The current input character type is sp. The current character is discarded, the last character of the last slice in the slice list is separated into a separate slice and added to the slice list, and the slice type is set to qs. The state is changed to sp.

[0147] 3. The current input character is of type ns. The current character is added to the slice list as a separate slice, and the slice type is set to ns. The state transitions to ns. Characters in the form of the symbol ".", excluding the first and last quotation marks.

[0148] 4. The current input character type is fs. The current character is added to the slice list as a separate slice, and the slice type is set to ns. The state transitions to ns. The input character is in the form of the symbol ".", which does not include the first and last quotation marks.

[0149] 5. The current input character type is os. The current character is added to the slice list as an independent slice, and the slice type is set to os. The state transitions to os. The input character is in the form of the symbol "<", which does not include the first and last quotation marks.

[0150] 6. The current input character type is cs. The current character is added to the slice list as an independent slice, and the slice type is set to cs. The state transitions to cs. The input character is in the form of the symbol "")", which does not include the first and last quotation marks.

[0151] 7. The current input character type is qs. Add the current character to the last slice in the slice list, set the slice type to nn, and transition to the nn state. For example, two consecutive quotation marks can be treated as ordinary characters.

[0152] By traversing the current input character to the last character through the above state transitions, the state transitions will terminate, and a slice list consisting of at least one slice will be obtained.

[0153] In step 130, at least one slice in the slice list is traversed, and a target slice is determined within that slice based on preset sentence segmentation rules. The preset sentence segmentation rules are used to indicate the sentence segmentation logic. The target slice is a slice containing sentence segmentation symbols and can also indicate the statement boundaries of the file to be identified.

[0154] The sentence boundary recognition method provided by this invention, through a finite state converter, can quickly split the file to be recognized into slices with minimal loop traversal of the string, thus improving recognition efficiency; combined with preset sentence segmentation rules, it can accurately determine the target slice, which not only saves time but also ensures recognition accuracy.

[0155] In some embodiments, the preset sentence segmentation rules include at least one of the following:

[0156] If no interval symbol exists in at least one slice, the slice of type ordinary clause character is determined as the target slice.

[0157] During the traversal of the slice list, if the interval symbol is empty in at least one slice, the slice of type ns (ordinary clause character type) is directly used as the clause point, that is, the slice is the target slice.

[0158] If no interval symbol exists in at least one slice, determine the next slice of the slice with the special clause character type;

[0159] If the first letter of the next slice is a capital letter, the slice with the special clause character type is identified as the target slice.

[0160] During the traversal of the slice list, if at least one slice contains an empty interval symbol, check the next slice of type fs (a special clause character type). Specifically, check if the first letter of the next slice containing letters is capitalized. Using a slice list data structure allows for efficient checking of slices near fs type slices.

[0161] The sentence boundary recognition method provided by this invention can efficiently traverse all slices by setting sentence segmentation rules, and thus quickly determine the target slice.

[0162] In some embodiments, the preset sentence segmentation rules include at least one of the following:

[0163] If a segment contains a range symbol, the segment with a type of ordinary clause character or special clause character is identified as a non-target segment.

[0164] If no interval symbol exists in at least one slice, determine the slice preceding the slice of the special clause character type;

[0165] If the characters in the previous slice meet the preset conditions, the slice with special clause character type will be identified as a non-target slice;

[0166] The preset conditions include at least one of the following:

[0167] The character fragment of the previous slice consists of an uppercase letter and at least one lowercase letter;

[0168] The character fragment of the previous slice contains at most two numbers;

[0169] The character fragments of the previous slice consisted of a single letter.

[0170] During the traversal of the slice list, if the interval identifier in at least one slice is not empty (regardless of how many levels of nesting the intervals), all slices of type ns (ordinary clause character type) or type fs (special clause character type) traversed in that slice are not treated as clause points, i.e., they are non-target slices.

[0171] During the traversal of the slice list, if the interval symbol is empty in at least one slice, the slice of type fs (a special clause character type) is processed as follows:

[0172] First, determine the slice preceding the slice of the special clause character type, and then determine whether the preceding slice meets the preset conditions.

[0173] If the character segment in the previous slice consists of an uppercase letter followed by one to three lowercase letters, this slice is not treated as a clause break; that is, the slice of this special clause character type is a non-target slice. For example, handling abbreviations such as "Dr." and "May."

[0174] If the character fragment in the previous slice consists of 1 to 2 digits, this slice is not treated as a clause break; that is, the slice of this special clause character type is a non-target slice. For example, this applies to handling item numbers such as "1." and "2.".

[0175] If the character fragment in the previous slice consists of a single uppercase letter, this slice is not treated as a clause break; that is, the slice of this special clause character type is a non-target slice. For example, this applies to handling item numbers such as "a." and "A."

[0176] It should be noted that this preset sentence segmentation rule may inadvertently cause sentences that should be segmented to not be segmented. However, in practice, not segmenting sentences has less impact on subsequent translation work than segmenting them incorrectly.

[0177] In some embodiments, the string corresponding to the text to be identified is input into a finite-state converter for splitting to obtain at least one slice, including:

[0178] If the first slice of the open interval symbol type is obtained, push the first slice onto the target stack;

[0179] If a second slice of the closed interval symbol type is obtained, the second slice is paired with the top element of the target stack;

[0180] If the first slice is successfully paired with the top element of the stack, a pop operation is performed on the top element of the stack.

[0181] During the traversal of the slice list, when the first slice of type os (open interval symbol type) is encountered, it can be directly pushed onto the target stack. The target stack is used to store interval symbols and quotation marks.

[0182] When encountering a second slice of type os (closed interval symbol type), check if the current stack top is the paired symbol corresponding to the second slice.

[0183] If yes, pop one element from the stack, i.e., perform a pop operation on the top element of the stack; otherwise, push the second slice onto the target stack.

[0184] In some embodiments, after obtaining a first slice of open interval symbol type and pushing the first slice onto the target stack, the method further includes:

[0185] If a third slice of type quotation mark character is obtained, the third slice is paired with the top element of the target stack;

[0186] If it is confirmed that the third slice is successfully paired with the top element of the stack, a pop operation is performed on the top element of the stack.

[0187] If it is determined that the third slice cannot be paired with the top element of the stack, the third slice is pushed onto the target stack.

[0188] During the traversal of the slice list, when a third slice of type qs (quote symbol type) is encountered, check whether the current top of the target stack is the matching symbol corresponding to the third slice.

[0189] If yes, pop an element from the stack, i.e., perform a pop operation on the top element of the stack; otherwise, push the third slice onto the target stack.

[0190] The following example of processing English sentences illustrates the embodiments provided by this invention.

[0191] For example: He's one of my best friends. Can't you (control) your children?

[0192] The string is traversed sequentially according to the rules.

[0193] For H:

[0194] Current state: init

[0195] Current slice list: []

[0196] Current slice: ()

[0197] Current input character: (H, nn)

[0198] State transition: nn

[0199] For e:

[0200] Current state: nn

[0201] Current slice list: []

[0202] Current slice: (H,nn)

[0203] Current input character: (e, nn)

[0204] State transition: nn

[0205] for':

[0206] Current state: nn

[0207] Current slice list: []

[0208] Current slice: (He, nn)

[0209] Current input character: (', qs)

[0210] State transition: qs

[0211] For the space symbol (the space between He's and one):

[0212] Current state: nn

[0213] Current slice list: []

[0214] Current slice: (He's, nn)

[0215] Current input character: (, sp)

[0216] State transition: sp

[0217] For o:

[0218] Current status: sp

[0219] Current slice list: [He's]

[0220] Current slice: ()

[0221] Current input character: (o, nn)

[0222] State transition: nn

[0223] For n:

[0224] Current state: nn

[0225] Current slice list [(He's, nn)]

[0226] Current slice: (o, nn)

[0227] Current input character: (n, nn)

[0228] State transition: nn

[0229] For e:

[0230] Current state: nn

[0231] Current slice list: [(He's, nn)]

[0232] Current slice: (on, nn)

[0233] Current input character: (e, nn)

[0234] State transition: nn

[0235] For the space symbol (the space between one and of):

[0236] Current state: nn

[0237] Current slice list: [(He's, nn)]

[0238] Current slice: (one, nn)

[0239] Current input character: (,nn)

[0240] State transition: sp

[0241] For o:

[0242] Current status: sp

[0243] Current slice list: [(He's, nn), (one, nn)]

[0244] Current slice: ()

[0245] Current input character: (o, nn)

[0246] State transition: nn

[0247] And so on, processing them sequentially according to the rules.

[0248] The sentence boundary recognition method provided by this invention processes standard input text sentences with minimal loop traversal of the string through a series of rules, resulting in very high processing efficiency.

[0249] The statement boundary recognition device provided by the present invention is described below. The statement boundary recognition device described below can be referred to in correspondence with the statement boundary recognition method described above.

[0250] Figure 2 This is a schematic diagram of the structure of the statement boundary recognition device provided by the present invention. (Refer to...) Figure 2 The statement boundary recognition device provided by the present invention includes: an acquisition module 210, a splitting module 220, and a determination module 230.

[0251] The acquisition module 210 is used to acquire the text to be recognized;

[0252] The splitting module 220 is used to input the string corresponding to the text to be identified into a finite state converter for splitting to obtain at least one slice. The slice is used to indicate the character fragments in the string and the slice type corresponding to the character fragments. The state in the finite state converter is constructed based on the character type.

[0253] The determination module 230 is used to determine a target slice in the at least one slice based on a preset sentence segmentation rule, wherein the target slice is used to indicate the sentence boundaries of the text to be identified.

[0254] The sentence boundary recognition device provided by this invention, through a finite state converter, can quickly split the file to be recognized into slices with minimal loop traversal of the string, thereby improving recognition efficiency; combined with preset sentence segmentation rules, it can accurately determine the target slice, which not only saves time but also ensures recognition accuracy.

[0255] In some embodiments, the character type includes at least one of the following:

[0256] Ordinary characters, including characters that do not affect clause segmentation;

[0257] Clause-segmenting characters, including ordinary clause-segmenting characters and special clause-segmenting characters. Ordinary clause-segmenting characters include characters that only have clause-segmenting function, while special clause-segmenting characters include all characters with clause-segmenting function except for the ordinary clause-segmenting characters.

[0258] Interval symbols, the interval symbols including at least one type of brackets, wherein an open interval symbol in the interval symbol is used to indicate the left half of the interval symbol, and a closed interval symbol in the interval symbol is used to indicate the right half of the interval symbol;

[0259] Space character;

[0260] Quotation characters;

[0261] The slice type includes at least one of the following: ordinary character type, ordinary clause character type, special clause character type, open interval symbol type, closed interval symbol type, space character type, and quotation mark character type.

[0262] In some embodiments, the preset sentence segmentation rules include at least one of the following:

[0263] If the interval symbol is not present in at least one slice, the slice of type ordinary clause character is determined as the target slice;

[0264] If the interval symbol is not present in at least one slice, determine the next slice of the slice with the special clause character type;

[0265] If the first letter of the latter slice is a capital letter, the slice of the special clause character type is determined as the target slice.

[0266] In some embodiments, the preset sentence segmentation rules include at least one of the following:

[0267] If the interval symbol is present in at least one slice, the slice whose type is the ordinary clause character type or the special clause character type is determined as a non-target slice;

[0268] If the interval symbol is not present in at least one slice, determine the slice preceding the slice of the special clause character type;

[0269] If the characters in the previous slice meet the preset conditions, the slice of the special clause character type is determined as a non-target slice;

[0270] The preset conditions include at least one of the following:

[0271] The character fragment of the previous slice includes an uppercase letter and at least one lowercase letter;

[0272] The character fragment of the previous slice includes at most two numbers;

[0273] The character fragments of the previous slice consist of a single letter.

[0274] In some embodiments, the splitting module is further configured to:

[0275] Upon obtaining the first slice of the open interval symbol type, push the first slice onto the target stack;

[0276] In the case of obtaining a second slice of the closed interval symbol type, the second slice is paired with the top element of the target stack;

[0277] If it is determined that the first slice is successfully paired with the top element of the stack, a pop operation is performed on the top element of the stack.

[0278] In some embodiments, after obtaining a first slice of the open interval symbol type and pushing the first slice onto the target stack, the method further includes:

[0279] If a third slice of the quoted character type is obtained, the third slice is paired with the top element of the target stack;

[0280] If it is determined that the third slice is successfully paired with the top element of the stack, a pop operation is performed on the top element of the stack.

[0281] If it is determined that the third slice cannot be paired with the top element of the stack, the third slice is pushed onto the target stack.

[0282] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a statement boundary recognition method, which includes:

[0283] Obtain the text to be recognized;

[0284] The string corresponding to the text to be identified is input into a finite state converter for splitting to obtain at least one slice. The slice is used to indicate the character fragments in the string and the slice type corresponding to the character fragments. The state in the finite state converter is constructed based on the character type.

[0285] Based on preset sentence segmentation rules, a target slice is determined in the at least one slice, and the target slice is used to indicate the sentence boundaries of the text to be identified.

[0286] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0287] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the statement boundary recognition method provided by the above methods, the method comprising:

[0288] Obtain the text to be recognized;

[0289] The string corresponding to the text to be identified is input into a finite state converter for splitting to obtain at least one slice. The slice is used to indicate the character fragments in the string and the slice type corresponding to the character fragments. The state in the finite state converter is constructed based on the character type.

[0290] Based on preset sentence segmentation rules, a target slice is determined in the at least one slice, and the target slice is used to indicate the sentence boundaries of the text to be identified.

[0291] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the statement boundary recognition method provided by the methods described above, the method comprising:

[0292] Obtain the text to be recognized;

[0293] The string corresponding to the text to be identified is input into a finite state converter for splitting to obtain at least one slice. The slice is used to indicate the character fragments in the string and the slice type corresponding to the character fragments. The state in the finite state converter is constructed based on the character type.

[0294] Based on preset sentence segmentation rules, a target slice is determined in the at least one slice, and the target slice is used to indicate the sentence boundaries of the text to be identified.

[0295] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0296] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0297] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying statement boundaries, characterized in that, include: Obtain the text to be recognized; The string corresponding to the text to be identified is input into the finite state converter. The finite state converter traverses the string corresponding to the text to be identified and splits the string to obtain at least one slice. The slice is used to indicate the character fragment in the string and the slice type corresponding to the character fragment. The state in the finite state converter is constructed based on the character type. Based on preset sentence segmentation rules, a target slice is determined in the at least one slice, and the target slice is used to indicate the sentence boundaries of the text to be identified; The character type includes at least one of the following: Ordinary characters, including characters that do not affect clause segmentation; Clause-segmenting characters, including ordinary clause-segmenting characters and special clause-segmenting characters. Ordinary clause-segmenting characters include characters that only have clause-segmenting function, while special clause-segmenting characters include all characters with clause-segmenting function except for the ordinary clause-segmenting characters. Interval symbols, the interval symbols including at least one type of brackets, wherein an open interval symbol in the interval symbol is used to indicate the left half of the interval symbol, and a closed interval symbol in the interval symbol is used to indicate the right half of the interval symbol; Space character; Quotation characters; The slice type includes at least one of the following: ordinary character type, ordinary clause character type, special clause character type, open interval symbol type, closed interval symbol type, space character type, and quotation mark character type; The preset sentence segmentation rules include at least one of the following: If the interval symbol is not present in at least one slice, the slice of type ordinary clause character is determined as the target slice; If the interval symbol is not present in at least one slice, determine the next slice of the slice with the special clause character type; If the first letter of the latter slice is a capital letter, the slice of the special clause character type is determined as the target slice.

2. The statement boundary recognition method according to claim 1, characterized in that, The preset sentence segmentation rules include at least one of the following: If the interval symbol is present in at least one slice, the slice whose type is the ordinary clause character type or the special clause character type is determined as a non-target slice; If the interval symbol is not present in at least one slice, determine the slice preceding the slice of the special clause character type; If the characters in the previous slice meet the preset conditions, the slice of the special clause character type is determined as a non-target slice; The preset conditions include at least one of the following: The character fragment of the previous slice includes an uppercase letter and at least one lowercase letter; The character fragment of the previous slice includes at most two numbers; The character fragments of the previous slice consist of a single letter.

3. The statement boundary recognition method according to claim 1, characterized in that, The step of inputting the string corresponding to the text to be identified into a finite state converter for splitting to obtain at least one slice includes: Upon obtaining the first slice of the open interval symbol type, push the first slice onto the target stack; In the case of obtaining a second slice of the closed interval symbol type, the second slice is paired with the top element of the target stack; If it is determined that the first slice is successfully paired with the top element of the stack, a pop operation is performed on the top element of the stack.

4. The statement boundary recognition method according to claim 3, characterized in that, After obtaining the first slice of the open interval symbol type and pushing the first slice onto the target stack, the method further includes: If a third slice of the quoted character type is obtained, the third slice is paired with the top element of the target stack; If it is determined that the third slice is successfully paired with the top element of the stack, a pop operation is performed on the top element of the stack. If it is determined that the third slice cannot be paired with the top element of the stack, the third slice is pushed onto the target stack.

5. A statement boundary recognition device, characterized in that, include: The acquisition module is used to acquire the text to be recognized; The splitting module is used to input the string corresponding to the text to be recognized into the finite state converter, and to traverse the string corresponding to the text to be recognized through the finite state converter to split the string and obtain at least one slice. The slice is used to indicate the character fragments in the string and the slice type corresponding to the character fragments. The state in the finite state converter is constructed based on the character type. The determination module is used to determine a target slice in the at least one slice based on a preset sentence segmentation rule, wherein the target slice is used to indicate the sentence boundaries of the text to be identified; The character type includes at least one of the following: Ordinary characters, including characters that do not affect clause segmentation; Clause-segmenting characters, including ordinary clause-segmenting characters and special clause-segmenting characters. Ordinary clause-segmenting characters include characters that only have clause-segmenting function, while special clause-segmenting characters include all characters with clause-segmenting function except for the ordinary clause-segmenting characters. Interval symbols, the interval symbols including at least one type of brackets, wherein an open interval symbol in the interval symbol is used to indicate the left half of the interval symbol, and a closed interval symbol in the interval symbol is used to indicate the right half of the interval symbol; Space character; Quotation characters; The slice type includes at least one of the following: ordinary character type, ordinary clause character type, special clause character type, open interval symbol type, closed interval symbol type, space character type, and quotation mark character type; The preset sentence segmentation rules include at least one of the following: If the interval symbol is not present in at least one slice, the slice of type ordinary clause character is determined as the target slice; If the interval symbol is not present in at least one slice, determine the next slice of the slice with the special clause character type; If the first letter of the latter slice is a capital letter, the slice of the special clause character type is determined as the target slice.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the statement boundary recognition method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the statement boundary recognition method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the statement boundary recognition method as described in any one of claims 1 to 4.

Images