System and method for Chinese punctuation recovery using sub-character information

Abstract

A system and method for predicting punctuation of text. The text is a logographic language having characters. Most characters contain sub-characters representing the meaning of the characters. The text lacks punctuation. The method includes: receiving the text; providing a sub-character encoding based on a sub-character input editor IME so that each character in the logographic language corresponds to a specific sub-character code; encoding the text using the sub-character encoding to obtain a sub-character code; generating a segment from the sub-character code; representing the segment by a segment vector; and placing the segment vector in a neural network to obtain punctuation of the text. The method can be implemented using a computing device.

Description

[0001] Cross-references

[0002] Several references are cited and discussed in the description of the present disclosure, which may include patents, patent applications, and various publications. The citation and / or discussion of such references is provided only to clarify the description of the present disclosure, and does not admit that any such reference is "prior art" to the disclosure described herein. All references cited in the "References" section or discussed in this specification are incorporated herein by reference in their entirety, and to the same extent as if each reference was incorporated by reference individually. Technical Field

[0003] The present disclosure relates to the field of punctuation recovery, and in particular, to a system and method for recovering punctuation of a Chinese text using sub-character information of Chinese characters. Background Art

[0004] The background description provided here is to generally present the context of the present disclosure. Within the scope of the description in this background section, the work of the inventors, as well as the description that may not qualify as prior art at the time of application, are neither explicitly nor implicitly admitted as prior art for the present disclosure.

[0005] In the current social media and mobile era, text without punctuation is common. For example, Audio Speech Recognition (ASR) is a common technology used by computing devices to recognize speech. The speech can be communicated through mobile applications or left as phone messages. ASR can convert speech into text. However, the translated text usually lacks punctuation. In another example, social media including SMS and Weibo contain informal texts, which often lack punctuation. Text without punctuation is poorly readable, resulting in a poor user experience. In addition, the performance of downstream applications will also be affected.

[0006] Furthermore, in e-commerce scenarios, texts in user-generated content are often informal. Such texts contain a large number of internet trending words, jargons and terms associated with domain names, as well as emoticons, etc. Due to the difficulties brought by these open vocabulary issues, recovering punctuation for informal written texts is more challenging.

[0007] A large number of studies have used language models, hidden Markov chains, conditional random fields (CRFs), and neural networks with vocabulary and acoustic features and word-level or character-level representations to predict punctuation. These methods with word-level or character-level models are insufficient in dealing with out-of-vocabulary (OOV) problems because they cannot provide word representations for out-of-vocabulary characters or words.

[0008] Therefore, for punctuation prediction of text, the presence of rare words or OOV words may lead to inaccurate predictions, and there is a need in the art to address the above-mentioned deficiencies and inadequacies. Summary of the invention

[0009] In some aspects, the present disclosure relates to a method for predicting punctuation of a text. The text is a logographic language having a plurality of characters, at least one of the plurality of characters includes a plurality of sub-characters representing the meaning of the at least one character, and the text lacks punctuation. In some embodiments, the length of the text corresponds to a small paragraph or a few sentences. In some embodiments, the text includes more than five words. In some embodiments, the text includes more than 10 words. In some embodiments, the text includes more than 15 words. In some embodiments, the text includes less than 500 words. In some embodiments, the text includes less than 100 words. In some embodiments, the method includes: receiving the text; providing a sub-character encoding based on a sub-character editor (or input method, or input method editor, abbreviated as IME) so that each character in the logographic language corresponds to a specific sub-character code; encoding the text using the sub-character encoding to obtain a sub-character code; generating a segment from the sub-character code; representing the segment by a segment vector; and placing the segment vector in a neural network to obtain the punctuation of the text.

[0010] In some embodiments, the language is Chinese, the characters are Chinese characters, and the sub-character input editor IME is Wubi or Stroke.

[0011] In some embodiments, the segments are generated using byte pair encoding.

[0012] In some embodiments, the segments are represented by vectors using word2vec. In some embodiments, the word2vec has a model architecture of skip-word or continuous bag of words. In some embodiments, the word2vec is trained by: providing a plurality of training texts; encoding the training texts using the sub-character encoding to obtain training sub-character codes; generating a segment vocabulary from the sub-character codes based on a predefined number of segments; and representing the segment vocabulary by a segment vector based on the context of the training text. In some embodiments, the language is Chinese, the number of characters is in the range of 6000 to 7000, and the predefined number of segments is in the range of 20000 to 100000. In some embodiments, the predefined number of segments is approximately 50000.

[0013] In some embodiments, the neural network is pre-trained in the following manner: providing a plurality of training texts with punctuation; encoding the training texts using the sub-character encoding to obtain training sub-character codes; generating training segments from the sub-character codes; representing the training segments by segment vectors; marking the training texts using the punctuation to obtain punctuation marks; generating predicted punctuation using the segment vectors by the neural network; and comparing the predicted punctuation with the punctuation marks to train the neural network.

[0014] In some embodiments, the neural network is a bidirectional long short-term memory BiLSTM. In some embodiments, the BiLSTM includes a multi-head attention layer.

[0015] In some embodiments, the method further includes: extracting the text from an e-commerce website; inserting the punctuation into the text to obtain text with punctuation; and replacing the text with the text with punctuation on the e-commerce website.

[0016] In some embodiments, the method further comprises: extracting audio from the video; processing the audio using audio speech recognition (ASR) to obtain the text; inserting the punctuation into the text to obtain text with punctuation; and adding the text with punctuation to the video. The added text with punctuation can be a subtitle of the video.

[0017] In some aspects, the present disclosure relates to a non-transitory computer-readable medium storing computer executable code. In some embodiments, when executed at a processor of a computing device, the computer executable code is configured to perform the above method.

[0018] In certain aspects, the present disclosure relates to a system for predicting punctuation of a text. The text is a logographic language having a plurality of characters, at least one of the plurality of characters includes a plurality of sub-characters representing the meaning of the at least one character, and the text lacks punctuation. The system includes a computing device having a processor and a storage device storing computer executable code, and when executed at the processor, the computer executable code is configured to: receive the text, provide a sub-character encoding based on a sub-character input editor IME, so that each character in the logographic language corresponds to a specific sub-character code; encode the text using the sub-character encoding to obtain a sub-character code; generate a segment from the sub-character code; represent the segment by a segment vector; and place the segment vector in a neural network to obtain the punctuation of the text.

[0019] In some embodiments, the language is Chinese, the characters are Chinese characters, the sub-character input editor IME is Wubi or Stroke, the computer executable code is configured to generate the sentence segment using byte pair encoding; and use word2vec to represent the sentence segment. In some embodiments, the neural network is a bidirectional long short-term memory BiLSTM.

[0020] In some embodiments, the word2vec is trained by providing a plurality of training texts; encoding the training texts using the sub-character encoding to obtain training sub-character codes; generating a segment vocabulary from the training sub-character codes based on a predefined number of segments; and representing the segment vocabulary by a segment vector based on the context of the training texts. In some embodiments, the language is Chinese, the number of characters is in the range of 6000 to 7000, and the predefined number of segments is in the range of 20000 to 100000. In some embodiments, the predefined number of segments is approximately 50000.

[0021] In some embodiments, the neural network is pre-trained by providing a plurality of training texts with punctuation marks; encoding the training texts using the sub-character encoding to obtain training sub-character codes; generating training segments from the training sub-character codes; representing the training segments by segment vectors; marking the training texts using the punctuation marks to obtain punctuation marks; generating predicted punctuation marks by the neural network using the segment vectors; and comparing the predicted punctuation marks with the punctuation marks to train the neural network. In some embodiments, the neural network is a bidirectional long short-term memory BiLSTM.

[0022] This technique helps to solve the problem of unknown words and insufficient vocabulary. In addition, the method encodes the similarity between characters, builds a powerful and more robust neural network model, and obtains comparable expressive power with fewer parameters with sub-character information.

[0023] These and other aspects of the present disclosure will become apparent from the following description of the preferred embodiments taken in conjunction with the following drawings and illustrations thereof, although changes and modifications therein may affect the spirit and scope of the novel concepts of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present disclosure will become more fully understood from the detailed description and the accompanying drawings. These drawings illustrate one or more embodiments of the present disclosure and, together with the written description, are used to explain the principles of the present disclosure. Wherever possible, the same reference numerals are used throughout the drawings to refer to the same or similar elements of the embodiments, wherein:

[0025] Figure 1A system for punctuation recovery according to certain embodiments of the present disclosure is schematically depicted.

[0026] Figure 2 The training of vector representations of segments according to certain embodiments of the present disclosure is schematically depicted.

[0027] Figure 3 The training of a punctuation prediction neural network according to certain embodiments of the present disclosure is schematically depicted.

[0028] Figure 4 Punctuation recovery according to certain embodiments of the present disclosure is schematically depicted.

[0029] Figure 5 Text paragraphs, their Wubi codes, and their sentence segments according to certain embodiments of the present disclosure are schematically depicted.

[0030] Figure 6 Text paragraphs, their stroke encodings, and their sentence segments according to certain embodiments of the present disclosure are schematically depicted.

[0031] Figure 7 The generation of punctuation marks for a paragraph of a text paragraph is schematically depicted according to certain embodiments of the present disclosure.

[0032] Figure 8 A bidirectional long short-term memory (BiLSTM) model for predicting punctuation from sentence segments according to certain embodiments of the present disclosure is schematically depicted.

[0033] Fig. 9 An improved BiLSTM model for predicting punctuation from sentence segments according to certain embodiments of the present disclosure is schematically depicted.

[0034] Fig.10 A method for training vector representation of a sentence segment according to certain embodiments of the present disclosure is schematically depicted.

[0035] Fig.11 A method for training a punctuation prediction neural network according to certain embodiments of the present disclosure is schematically depicted.

[0036] Fig.12 A method for punctuation recovery according to certain embodiments of the present disclosure is schematically depicted. DETAILED DESCRIPTION

[0037] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations therein will be apparent to those skilled in the art. Various embodiments of the present disclosure are now described in detail. With reference to the accompanying drawings, the same numerals indicate the same components throughout the views. Unless the context clearly dictates otherwise, the meanings of "one", "an" and "said" used in the description herein and throughout the claims include the plural. In addition, as used in the description and claims of the present disclosure, unless the context clearly dictates otherwise, the meaning of "in" includes "in" and "on". Furthermore, titles or subtitles may be used in the specification for the convenience of the reader, which does not affect the scope of the present disclosure. In addition, some of the terms used in this specification are more specifically defined below.

[0038] The terms used in this specification generally have their ordinary meanings in the art, in the context of this disclosure, and in the specific context in which each term is used. Certain terms used to describe the present disclosure are discussed below or elsewhere in the specification to provide practitioners with additional guidance on the description of the present disclosure. For convenience, certain terms may be highlighted, such as using italics and / or quotation marks. The use of highlighting has no effect on the scope and meaning of the term; in the same context, the scope and meaning of a term are the same whether it is highlighted or not. It is understood that the same thing can be expressed in more than one way. Therefore, alternative language and synonyms can be used for any one or more terms discussed herein, and there is no special meaning for whether the terms are elaborated or discussed in detail here. The present disclosure provides synonyms for certain terms. The use of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any term discussed herein, is illustrative only and in no way limits the scope and meaning of the present disclosure or any exemplary term. Similarly, the present disclosure is not limited to the various embodiments given in this specification.

[0039] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. In the event of a conflict, the present document, including definitions, will control.

[0040] As used herein, the terms "includes," "comprising," "carrying," "having," "containing," "involving," etc. should be understood as open-ended, meaning including but not limited to.

[0041] As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A or B or C), using a non-exclusive logical OR. It should be understood that one or more steps within a method can be executed in different order (or concurrently) without altering the principles of the present disclosure.

[0042] As used herein, the term "module" or "unit" may indicate belonging to or including an application specific integrated circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip. The term module or unit may include a memory (shared, dedicated, or group) that stores code executed by a processor.

[0043] The term "code" used herein may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class and / or an object. The term shared used above means that a single (shared) processor may be used to execute some or all of the code from multiple modules. In addition, some or all of the code from multiple modules may be stored in a single (shared) memory. The term group used above means that a group of processors may be used to execute some or all of the code from a single module. In addition, a group of memories may be used to store some or all of the code from a single module.

[0044] As used herein, the term "interface" generally refers to a communication tool or device used to perform data communication between components at an interaction point between components. Generally speaking, an interface can be applicable at both the hardware and software levels and can be a unidirectional or bidirectional interface. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. The components that communicate with the interface can be, for example, multiple components or peripheral devices of a computer system.

[0045] The present disclosure relates to computer systems. As shown in the accompanying drawings, computer components may include physical hardware components, which are shown as solid line blocks, and virtual software components, which are shown as dashed line blocks. It will be understood by those of ordinary skill in the art that, unless otherwise specified, these computer components may be implemented in the form of software, firmware or hardware components or a combination thereof, but are not limited to these forms.

[0046] The devices, systems, and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer programs may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0047] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0048] In some aspects, the present disclosure combines sub-character information with a sentence segment algorithm to address rare word or OOV problems and improve punctuation prediction accuracy. Additionally, by using sentence segments composed of sub-characters, the computational overhead is significantly reduced by decreasing the number of encoding sequences corresponding to the text. Further, the present disclosure encodes the similarity between characters to build a powerful and more robust language model and obtains comparable expressive power with fewer parameters with sub-character information.

[0049] The present disclosure creatively combines sub-characters into sentence segments for punctuation restoration. Sub-character information exists in a variety of logographic languages, such as Chinese, Egyptian hieroglyphs, Mayan glyphs, and their derivatives. In Chinese, the sub-character information of a Chinese character can include the strokes or radicals of a character. A stroke is the movement of a writing tool on a writing surface, and each character has a stroke order that contains multiple strokes in sequence. A radical includes one or several strokes. The radical of a character usually represents the smallest semantic unit, and different characters can share the same radical. For example, the radical "忄" is usually related to personal emotions and mental states. Characters containing "忄" include "忧", "恨", "怕", "惜", "忆", etc. When the radical "月" is on the left or bottom of a character, it is related to the human body, such as "脸", "肝", "胸", "臀", while when the radical "月" is on the right of a character, it is related to time, such as "期". The radical "扌" is related to the human hand and arm, and characters containing the radical "扌" are usually verbs. When the model of the present disclosure knows that a word or character containing the radical "扌" is used as a verb, the model is less likely to place these characters at the end of a sentence. The system of the present disclosure also uses the sentence segment algorithm to automatically build a customized vocabulary for the target system. After establishing the vocabulary, the present disclosure further trains the vector representation of the vocabulary using methods such as continuous bag-of-words or Skip Gram. Then, the vector representation is used as the input of a trained neural network model for predicting punctuation marks.

[0050] In some aspects, the present disclosure provides a system and method for predicting punctuation of text based on sub-character information. Figure 1 A system for punctuation prediction according to some embodiments of the present disclosure is schematically depicted. As Figure 1As shown, system 100 includes computing device 110. In some embodiments, computing device 110 can be a server computer, a cluster, a cloud computer, a general-purpose computer, a headless computer, or a special-purpose computer, which can predict punctuation. Computing device 110 can include, but is not limited to, processor 112, memory 114, and storage device 116. In some embodiments, computing device 110 can include other hardware components and software components (not shown) to perform their corresponding tasks. Examples of these hardware and software components can include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0051] Processor 112 may be a central processing unit (CPU) configured to control the operation of computing device 110. In some embodiments, processor 112 may execute an operating system (OS) or other application of computing device 110. In some embodiments, computing device 110 may have more than one CPU as a processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. Memory 114 may be a volatile memory, such as a random access memory (RAM), for storing data and information during operation of computing device 110. In some embodiments, memory 114 may be a volatile memory array. In some embodiments, computing device 110 may run on more than one processor 112 and / or more than one memory 114. Storage device 116 is a non-volatile data storage medium or device. Examples of storage device 116 may include flash memory, memory card, USB drive, solid state drive, or other types of non-volatile storage devices, such as hard disk drives, floppy disks, optical drives, or any other types of data storage devices. In some embodiments, computing device 110 may have more than one storage device 116. In certain embodiments, computing device 110 may also include remote storage device 116 .

[0052] The storage device 116 stores computer executable code. The computer executable code includes a punctuation recovery application 118, and optionally input data for training and predicting the punctuation recovery application 118. The punctuation recovery application 118 includes code or instructions that, when executed at the processor 112, predict or recover punctuation from text, where the text lacks punctuation. In some embodiments, the punctuation recovery application 118 may not be executable code, but may be in the form of circuitry corresponding to the functionality of the executable code. By providing circuitry rather than executable code, the speed of operation of the punctuation recovery application 118 is greatly improved. In some embodiments, such as Figure 1 As shown, the punctuation recovery application 118 includes a vector representation training module 120, a punctuation neural network training module 130, a punctuation prediction module 140, a function module 150, a user interface 170, and the like.

[0053] The vector representation training module 120 includes a sub-character encoding module 122, a segment generation module 124, and a vector representation module 126 for training the vector representation of the input text. Figure 2 A flowchart schematically depicting the training of a vector representation used as an input to a neural network in accordance with certain embodiments of the present disclosure is shown. As Figure 2 shown, when an input text A for vector representation training, such as a Chinese sentence, is provided, the sub-character encoding module 122 is configured to encode the input text A into sub-character encodings and send the sub-character encodings to the segment generation module 124. In certain embodiments, the sub-character encoding may be performed based on Wubi encoding, stroke encoding, or any other type of encoding that takes into account sub-characters of the characters. For input methods such as Wubi or stroke, keyboard combinations can be used as input, inputting codes corresponding in a computing device, which may correspond to multiple different characters, and the user can select one of the different characters as the final input. In the Wubi or stroke-based encoding of the present disclosure, the above process is reversed. Specifically, for each character input, the sub-character encoding module 122 provides a code, with each character corresponding to a determined code. Since there is a one-to-one correspondence between the character and the code, no manual interaction is required. Figure 5 The Wubi encodings of two Chinese sentences in accordance with certain embodiments of the present disclosure are shown. For the sentences in the example, there are corresponding Wubi encodings, where each character is encoded by its complete Wubi code. As Figure 5 shown, the full codes corresponding to the first three characters "不知道" of the first sentence are "gii", "tdkg", "uthp" respectively. Note that Figure 5 the punctuation in the sentence is shown in the Wubi encoding, making the correspondence between the example characters and their Wubi encodings obvious. However, the sub-character encoding module 122 may only encode the characters, so Figure 5 the punctuation symbols in the Wubi encoding in Figure 6 are non-existent. In addition, the sub-character encoding module 122 may preserve the spaces between the characters during the encoding process. In certain embodiments, the spaces are encoded with the same special code. In certain embodiments, the sub-character encoding module 122 is configured to encode the input text using stroke encoding. In stroke encoding, each Chinese character includes one or more consecutive strokes, and the strokes are divided into five types corresponding to the numbers 1, 2, 3, 4, 5, and then each Chinese character can be represented by a consecutive number composed of some of these five numbers. As

[0054] The sentence segment generation module 124 is configured to generate a sentence segment after receiving the sub-character encoding, and send the sentence segment to the vector representation module 126. In some embodiments, the sentence segment generation module 124 is configured to generate the sentence segment using byte pair encoding. Specifically, the sub-character encoding is a continuous encoding without punctuation. For example, Figure 5 The Wubi encoding of the first sentence shown is "gii tdkg uthp etnh bnh ewgi eukq hci ewgi tvey fggh jfd dmjd lpk whj utekkkf yukq jeg bnhn imcy def", which is converted into sentence segments. Depending on the parameters of the byte pair encoding, the separation between sentence segments may not fall between characters. In other words, each unit of a sentence segment may contain the Wubi encoding of a subcharacter, such as "the first subcharacter of character one"; some subcharacters, such as "the first and second subcharacters of character one"; a character, such as "character one"; two or more adjacent characters, such as "character one, character two, and character three"; one or more characters and subcharacters, such as "character one, character two, and the first subcharacter of character three"; or one or more characters with one or more subcharacters, such as "the last subcharacter of character one, the character of character two, the first subcharacter of character three". For example, for a sentence containing three characters, character one has three sub-characters, character two has four sub-characters, and character three has four sub-characters. Then possible segments may include, for example: "sub-character one of character one", "sub-character two of character one", ..., "sub-character four of character three", "character one", "character two", "character three", "character one-character two", "character two-character three", "character one-character two-character three", "sub-character one of character one-character two-character three", "sub-character one of character one-character two-character three", "sub-character one of character one-character two-character three", "sub-character one of character one-character two-character three" encoding. In some embodiments, the total number of generated segments is predetermined, and the segments with high frequency of occurrence in the entire training data set are selected as segments. In some embodiments, although Wubi defines approximately 6,000 to 7,000 Chinese characters, the present disclosure can define the total number of generated segments within the range of 20,000 to 100,000, for example, as 50,000. In some embodiments, the generated segments are collectively referred to as the vocabulary of the punctuation recovery application 118. Please note that the vocabulary itself is not predefined, but is automatically generated by the segment generation module 124. The segments in the vocabulary may include sub-characters, characters, combinations of sub-characters, combinations of characters, and combinations of sub-characters and characters. Through segment generation, information hidden in the sub-characters of Chinese characters can be extracted and interpreted.

[0055] The vector representation module 126 is configured to, after receiving the encoded, generated segments, train the vector representation of the segments. The training of the vector representation module 126 requires a large amount of input text data so that the distance between the vectors represents the contextual relationship between the corresponding segments. In some embodiments, the input text can be selected from the data on a specific topic. For example, the input text related to the product category in the e-commerce website can be retrieved to train the vector representation, and the trained punctuation recovery application 118 can be used to predict the punctuation of the text related to the product category. In some embodiments, a wide range of input texts are selected to train the vector representation so that the trained punctuation recovery application 118 can be used in various scenarios. In some embodiments, the vector representation module 126 is a word2vec model, which can be trained using skip words or continuous bag of words (CBOW). In some embodiments, the dimension of the vector is in the range of about 50 to 1000. In some embodiments, the dimension of the vector is 300. After training, each segment in the vocabulary is represented by a vector, and the distance between the vectors represents the relationship between the corresponding segments in the context. In some embodiments, the vector representation is a key-value format. In some embodiments, since the segments and their vector representations correspond one-to-one to each other, the overall segments and the overall vector representations can both be referred to as a vocabulary or dictionary of the segments.

[0056] In some embodiments, varying the vocabulary size of a segment explores the trade-off between computational efficiency and the ability to capture similarities between different characters.

[0057] In some embodiments, the punctuation recovery application 118 is configured to use a glyph vector representation for the input text A instead of using the sub-character encoding module 122 and the segment generation module 124. In some embodiments, the punctuation recovery application 118 is configured to use a glyph vector representation in combination with sub-character encoding.

[0058] The punctuation neural network training module 130 is configured to train the punctuation neural network after the vector representation training module 120 completes the vector representation training. Figure 1 The punctuation neural network training module 130 includes a training input generator 132 , a training token generator 134 and a neural network 136 . Figure 3 Schematically depicts a flow chart for training a neural network 136 according to certain embodiments of the present disclosure. Figure 3 As shown, input text B is provided for training the neural network 136 for punctuation prediction. Figure 3 The input text B in can be Figure 2The input text A in the training input generator 132 may be the same as or different from the input text A in the training input generator 132. The input text B may be, for example, a Chinese sentence. The training input generator 132 is configured to, when the input text B is available, instruct the sub-character encoding module 122 to encode the input text B into a sub-character encoding (or a sub-character code), instruct the segment generation module 124 to generate a segment according to the sub-character encoding, instruct the trained vector representation module 126 to provide a vector representation of the generated segment, and send the vector representation to the neural network 136. The segment generation module 124 is configured to generate segments based on the segment vocabulary constructed during the training of the vector representation module 126, where each sentence may correspond to a group of segments with a high occurrence rate in the vocabulary.

[0059] Meanwhile, the training token generator 134 is used to automatically generate tokens for the input text B, and send the tokens to the neural network 136. In some embodiments, the present disclosure defines punctuation recovery as a sequence-to-sequence labeling problem. Figure 7 Schematically depicts the generation of annotation tokens from input text. Figure 7 As shown, when input text such as one or several sentences is available, the training token generator 134 is configured to annotate characters (tokens) as a set of symbols. If a character is followed by another character, the character is annotated with the symbol "O", and if the character is followed by a punctuation mark, it is labeled based on the punctuation mark after the character. For example, if the character is followed by a question mark, the character is annotated as "Q", if the character is followed by a period, the character is annotated as "P", and if the character is followed by a comma, the character is annotated as "C".

[0060] Return to reference Figure 3 , the neural network 136 is configured to, after receiving the vector representation of the input text B from the vector representation module 126 and the corresponding token from the training token generator 134, predict the presence of punctuation marks based on the vector representation of the sentence segment, compare the predicted punctuation marks and their positions with the tokens, use a loss function to penalize incorrect token predictions and encourage correct token predictions, and obtain a trained neural network 136 after multiple rounds of training using the input text B. Each round of training can be performed using a certain number of sentences in the input text B, and the training using a certain number of sentences can be performed multiple times to reach convergence.

[0061] Figure 8 Schematically depicts an exemplary model structure of a neural network 136 according to certain embodiments of the present disclosure. Figure 8As shown, neural network 136 is a bidirectional long short-term memory (BidirectionalLong Short-term Memory, BiLSTM). When a segment sequence is available, the segment vector is input to BiLSTM. In some embodiments, the σ function uses a softmax function to output predicted tags. In some embodiments, the LSTM component can be changed to other cyclic computing units, such as a gated recurrent unit (Gated Recurrent Unit, GRU). In some embodiments, the LSTM layer can have multiple layers instead of just one layer. The multi-layer of LSTM can help capture the syntactic and semantic information of the input text. However, it will generate more computational overhead.

[0062] Fig. 9 Schematically depicts another exemplary model structure of the neural network 136 according to certain embodiments of the present disclosure. Fig. 9 As shown, the present disclosure uses a Bi-LSTM recurrent network with multi-head attention followed by a softmax function. Attention is good at capturing interactions between sequences.

[0063] The punctuation prediction module 140 is configured to predict punctuation of the input text after the neural network 136 is trained. Figure 4 Schematically depicts a flow chart of punctuation prediction according to certain embodiments of the present disclosure. Figure 4 As shown, an input text C for punctuation prediction is provided. For example, the input text C can be a text comprising several sentences. For example, the length of the input text C can be in the range of from five characters to several hundred characters. In certain embodiments, the input text C includes 10 to 50 characters. The punctuation prediction module 140 is used to indicate that the sub-character encoding module 122 encodes the input text C into sub-character encoding (or sub-character code) when the input text C is available, indicate that the sentence segment generation module 124 generates a sentence segment from the sub-character encoding, indicate that the trained vector representation module 126 provides the vector representation of the generated sentence segment, and indicate that the trained neural network 136 predicts punctuation from the sentence segment. The generated sentence segment is based on the trained sentence segment model. The punctuation prediction module 140 is also configured to, after the neural network 136 performs punctuation prediction, add the predicted punctuation to the original input text C to form a text C with punctuation.

[0064] Functional module 150 may be stored in computing device 110 or any other computing device in communication with computing device 110. Functional module 150 is configured to apply punctuation recovery application 118 to perform certain functions. In some embodiments, the function is to predict punctuation for social media messages that lack punctuation, and functional module 150 is configured to instruct sub-character encoding module 122 to encode media messages into sub-character encodings, instruct segment generation module 124 to generate segments from sub-character encodings, instruct trained vector representation module 126 to provide vector representations of generated segments, and instruct trained neural network 136 to predict punctuation based on segments. The predicted punctuation can be added back to the media message, and the media message with punctuation can be displayed to the user or stored for use in other applications.

[0065] In some embodiments, the function is to predict punctuation for text recognized from audio, wherein the recognized text does not include punctuation. Punctuation is predicted for the recognized text by the method described above, and the predicted punctuation is added to the recognized text. In some embodiments, the recognized text is derived from a movie or video, punctuation is predicted for the recognized text by the method described above, the predicted punctuation is added to the recognized text, and the recognized text with the predicted punctuation is added to the movie or video as subtitles.

[0066] In some embodiments, the function is to add punctuation to a message entered by a user in an e-commerce platform or a mobile phone, but the message does not contain punctuation. The function can predict the punctuation of the message and add the punctuation to the edited message so that the user can confirm the correctness of the punctuation and publish the message to the e-commerce platform or send a mobile message after confirmation.

[0067] User interface 160 is configured to provide a user interface or graphical user interface in computing device 110. In some embodiments, a user or administrator of the system can configure parameters for computing device 110, such as the size or size range of segments.

[0068] Fig.10 Schematically depicts a method for vector representation training according to certain embodiments of the present disclosure. In certain embodiments, Fig.10 The method 1000 shown may be performed as follows. Figure 1 It should be noted that, unless otherwise specified in the present disclosure, the steps of the above method can be arranged in different orders, and are therefore not limited to Fig.10 In some embodiments, Fig.10 The process shown in corresponds to Figure 2 Flowchart shown.

[0069] In step 1002, training input text is provided. The training text input includes a large number of text data sets. The text data sets are in a logographic language, such as Chinese. Each text data set may include one or more sentences and punctuation marks.

[0070] At step 1004, for each input text data set, the sub-character encoding module 122 converts the text data set into a sub-character encoding and sends the sub-character encoding to the sentence segment generation module 124. In some embodiments, the data set is Chinese and the sub-character encoding is Wubi or stroke.

[0071] At step 1006, after receiving the sub-character encoding, the segment generation module 124 generates a segment from the encoding and sends the generated segment to the vector representation module 126. In some embodiments, the generation of the segment is performed using byte pair encoding. In some embodiments, the segment generation module 124 defines the size of the segment vocabulary. In some embodiments, for Chinese, the size of the vocabulary is in the range of 20,000 to 100,000. In some embodiments, the size of the vocabulary is about 50,000.

[0072] At step 1008, after receiving the generated segments from the training dataset, the vector representation module 126 trains the vector representation of the generated segments. In some embodiments, the training of the vector representation is performed using skipword or continuous bag of words.

[0073] After training the vector representation, we can train a neural network to predict punctuation. Fig.11 Schematically depicts a method for training a punctuation prediction neural network according to certain embodiments of the present disclosure. In certain embodiments, such as Fig.11 The method 1100 shown may be performed as follows. Figure 1 It should be noted that, unless otherwise specified in the present disclosure, the steps of the above method can be arranged in different orders, and are therefore not limited to Fig.11 In some embodiments, Fig.11 The steps shown in correspond to Figure 3 The flowchart shown.

[0074] In step 1102, training input text is provided. The training input text includes a large number of text data sets. The text data sets are in a logographic language, such as Chinese. Each text data set may include one or more sentences and punctuation marks.

[0075] In step 1104, the training input generator 132 instructs the sub-character encoding module 122 to convert the text data set into sub-character encoding, instructs the sentence segment generation module 124 to generate a sentence segment from the sub-character encoding, instructs the trained vector representation module 126 to retrieve the corresponding vector representation of the generated sentence, and sends the vector representation to the neural network 136.

[0076] At step 1106, the training tokenizer 134 automatically creates tokens for the text dataset and sends the tokens to the neural network 136. In some embodiments, for each character in each text dataset, a specific character followed by a character is labeled with the same indicator to indicate that the specific character is not preceded by a punctuation mark, and another specific character followed by a punctuation mark is labeled with an indicator to indicate the type of punctuation mark following the other specific character.

[0077] At step 1108, after receiving the vector representation of the training text dataset and its corresponding punctuation marks, the neural network 136 predicts the punctuation marks from the vector representation, compares the predicted punctuation marks with the marks, and updates its parameters based on the comparison. The neural network 136 can be well trained through multiple rounds of training using a large training dataset.

[0078] After the punctuation prediction neural network is trained, the application of the present disclosure can be used to perform punctuation prediction. Fig.12 Schematically depicts a method for predicting punctuation marks from text input according to certain embodiments of the present disclosure. In certain embodiments, Fig.12 The method 1200 shown may be performed as follows. Figure 1 It should be noted that, unless otherwise specified in the present disclosure, the steps of the above method can be arranged in different orders, and are therefore not limited to Fig.12 In some embodiments, Fig.12 The process shown in corresponds to Figure 4 Flowchart shown.

[0079] In step 1202, an input text for punctuation prediction is provided. The input text is a logographic language such as Chinese, and the input text does not contain punctuation.

[0080] In step 1204, the punctuation prediction module 140 instructs the sub-character encoding module 122 to convert the input text into sub-character encodings, instructs the sentence segment generation module 124 to generate a sentence segment according to the sub-character encodings, instructs the trained vector representation module 126 to retrieve the corresponding vector representation of the generated sentence, and sends the vector representation to the trained neural network 136. In some embodiments, when there is a novel character in the input text, and the sub-character encoding module 122 can identify only one or several sub-characters from the novel character, the sub-character encoding module 122 can take the one or several sub-characters as representatives of the entire novel character. For example, if the novel character has only one identifiable sub-character, and the sub-character corresponds to a code, then the code can be regarded as the code of the novel character.

[0081] At step 1206, after receiving the vector representation of the input text, the trained neural network 136 predicts punctuation from the vector representation. In some embodiments, when a segment ends with a sub-character, the predicted punctuation for the sub-character may be placed after the character containing the sub-character.

[0082] In some aspects, the present disclosure relates to applications that rely on punctuation recovery. In some embodiments, the applications include, for example, semantic parsing, question answering, text summarization, subtitling, and machine translation.

[0083] In some aspects, the present disclosure relates to a non-transitory computer-readable medium storing computer executable code. The code, when executed at the processor 112 of the computing device 110, can perform the methods 1000, 1100, and 1200 described above. In some embodiments, the non-transitory computer-readable medium can include, but is not limited to, any physical or virtual storage medium. In some embodiments, the non-transitory computer-readable medium can be implemented as Figure 1 The storage device 116 of the computing device 110 is shown.

[0084] In summary, certain embodiments of the present disclosure have the following beneficial advantages in particular. First, the method and system of the present disclosure take into account the sub-character information that is unique to logographic languages ​​such as Chinese, and use the sub-character information to predict whether there is a punctuation mark after a character. By adding the sub-character information of a character to the punctuation prediction model, the prediction accuracy is significantly improved. Furthermore, by combining sub-character encoding with sentence segment generation, the extraction of sub-characters can be effectively achieved. Through effective extraction, sub-character information is attributed to accurate punctuation prediction. Third, the number of sub-characters is small compared to the number of characters. When the text used for punctuation prediction includes unknown or novel characters, the novel character is likely to have one or several recognizable sub-character components. Although the prediction model does not know the meaning of the novel character, it can still make reasonable punctuation predictions based on the sub-character components of the novel character. Therefore, when encountering novel characters, the method and system perform well. Fourth, with the rapid development of social media or fashion, new usages of known characters may be created. Without knowing the exact meaning of the new usage, the method and system of the present disclosure can still make accurate punctuation predictions based on the sub-characters of the characters with the new usage.

[0085] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description only and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the above teachings.

[0086] The embodiments are selected and described in order to explain the principles of the present disclosure and its practical application, so that other persons skilled in the art can utilize the present disclosure and various embodiments and various modifications suitable for the intended specific use. Alternative embodiments will become apparent to those skilled in the art to which the present disclosure belongs without departing from the spirit and scope of the present disclosure. Therefore, the scope of the present disclosure is limited by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

Claims

1. A method for predicting punctuation of a text, comprising: receiving the text, wherein the text uses a logographic language having a plurality of characters, at least one of the plurality of characters includes a plurality of sub-characters representing a meaning of the at least one character, and the text lacks punctuation; Providing sub-character encoding based on a sub-character input editor IME so that each character in the logogram language corresponds to a specific sub-character code; Encoding the text using the sub-character encoding to obtain a sub-character code; Generating a sentence segment from the sub-character code, wherein the sentence segment includes a combination of a sub-character and a character, a combination of characters, and a combination of sub-characters; Representing the segment by a segment vector; and The sentence segment vector is placed in a neural network to obtain the punctuation of the text.

2. The method according to claim 1, wherein: The language is Chinese, the characters are Chinese characters, and the sub-character input editor IME is Wubi or Stroke.

3. The method according to claim 1, wherein: The segments are generated using byte pair encoding.

4. The method according to claim 1, wherein: The sentence segments are represented by vectors using word2vec, which has a skip-word or continuous bag-of-words model architecture.

5. The method according to claim 4, wherein: The word2vec is trained in the following way: Provide multiple training texts; Encoding the training text using the sub-character encoding to obtain a training sub-character code; generating a segment vocabulary from the sub-character codes based on a predefined number of segments; as well as The segment vocabulary is represented by segment vectors based on the context of the training text.

6. The method according to claim 5, wherein: The language is Chinese, the number of characters is in the range of 6,000 to 7,000, and the number of predefined segments is in the range of 20,000 to 100,000.

7. The method according to claim 1, wherein: The neural network is pre-trained in the following way: Provide multiple training texts with punctuation; Encoding the training text using the sub-character encoding to obtain a training sub-character code; generating a training segment from the sub-character codes; Representing the training segment by a segment vector; Marking the training text with the punctuation marks to obtain punctuation marks; generating predicted punctuation using the sentence segment vectors through the neural network; as well as The predicted punctuation is compared to the punctuation markers to train the neural network.

8. The method according to claim 1, wherein: The neural network is a bidirectional long short-term memory BiLSTM.

9. The method according to claim 1, further comprising: extracting said text from an e-commerce website; inserting the punctuation marks into the text to obtain a text with punctuation marks; as well as The text is replaced with the punctuated text on the e-commerce website.

10. The method according to claim 1, further comprising: Extract audio from video; Processing the audio using audio speech recognition (ASR) to obtain the text; inserting the punctuation marks into the text to obtain a text with punctuation marks; as well as The punctuated text is added to the video.

11. A system for predicting punctuation of text, wherein: The system includes a computing device including a processor and a storage device storing computer executable code, the computer executable code when executed at the processor is configured to: receiving the text, wherein the text uses a logographic language having a plurality of characters, at least one of the plurality of characters includes a plurality of sub-characters representing a meaning of the at least one character, and the text lacks punctuation; Providing sub-character encoding based on a sub-character input editor IME so that each character in the logogram language corresponds to a specific sub-character code; Encoding the text using the sub-character encoding to obtain a sub-character code; Generating a sentence segment from the sub-character code, wherein the sentence segment includes a combination of a sub-character and a character, a combination of characters, and a combination of sub-characters; Representing the segment by a segment vector; and The sentence segment vector is placed in a neural network to obtain the punctuation of the text.

12. The system according to claim 11, wherein: The language is Chinese, the characters are Chinese characters, the sub-character input editor IME is Wubi or Stroke, and the computer executable code is configured to: generating the segment using byte pair encoding; and The sentence segment is represented using word2vec.

13. The system according to claim 12, wherein: The word2vec is trained in the following way: Provide multiple training texts; Encoding the training text using the sub-character encoding to obtain a training sub-character code; generating a segment vocabulary from the training sub-character codes based on a predefined number of segments; as well as The segment vocabulary is represented by segment vectors based on the context of the training text.

14. The system according to claim 13, wherein: The language is Chinese, the number of characters is in the range of 6,000 to 7,000, and the number of predefined segments is in the range of 20,000 to 100,000.

15. The system according to claim 11, wherein: The neural network is pre-trained in the following way: Provide multiple training texts with punctuation; Encoding the training text using the sub-character encoding to obtain a training sub-character code; Generating a training segment from the training sub-character codes; Representing the training segment by a segment vector; Marking the training text with the punctuation marks to obtain punctuation marks; generating predicted punctuation using the sentence segment vectors through the neural network; as well as The predicted punctuation is compared to the punctuation markers to train the neural network.

16. The system of claim 15, wherein: The neural network is a bidirectional long short-term memory BiLSTM.

17. The system of claim 11, wherein: The neural network is pre-trained in the following way: Provide multiple training texts with punctuation; Encoding the training text using the sub-character encoding to obtain a training sub-character code; Generating a training segment from the training sub-character codes; Representing the training segment by a segment vector; Marking the training text with the punctuation marks to obtain punctuation marks; generating predicted punctuation using the sentence segment vectors through the neural network; as well as The predicted punctuation is compared to the punctuation markers to train the neural network.

18. A non-transitory computer readable medium storing computer executable code, wherein: The computer executable code, when executed at a processor of a computing device, is configured to: receiving a text, wherein the text uses a logographic language having a plurality of characters, at least one of the plurality of characters includes a plurality of sub-characters representing a meaning of the at least one character, and the text lacks punctuation; Providing sub-character encoding based on a sub-character input editor IME so that each character in the logogram language corresponds to a specific sub-character code; Encoding the text using the sub-character encoding to obtain a sub-character code; Generating a sentence segment from the sub-character code, wherein the sentence segment includes a combination of a sub-character and a character, a combination of characters, and a combination of sub-characters; Representing the segment by a segment vector; and The sentence segment vector is put into a neural network to obtain punctuation of the text.

19. The non-transitory computer readable medium of claim 18, wherein: The language is Chinese, the characters are Chinese characters, the neural network is a bidirectional long short-term memory BiLSTM, and the computer executable code is configured to: generating the segment using byte pair encoding; and The sentence segment is represented using word2vec.

20. The non-transitory computer readable medium of claim 19, wherein: The word2vec is trained in the following way: Provide multiple training texts; Encoding the training text using the sub-character encoding to obtain a training sub-character code; generating a segment vocabulary from the training sub-character codes based on a predefined number of segments; as well as The segment vocabulary is represented by segment vectors based on the context of the training text.

Images