A method, device and medium for accurately decoding Chinese real-time speech and punctuation

Abstract

The present invention belongs to the technical field of automatic speech recognition and text processing, and provides a method, device and medium for accurate decoding of punctuation in real-time Chinese speech. The present invention designs a dynamic text stream receiver based on a Chinese punctuation prediction model, which is continuously adjusted to include text with undetermined sentence boundaries and newly input text in a temporary storage area, adds punctuation marks by dynamically detecting sentence boundaries, and outputs them immediately after determining a complete sentence, thereby improving the accuracy of punctuation annotation in real-time text stream transcription and significantly improving the readability and semantic clarity of the text.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic speech recognition and text processing, and in particular, relates to a method, device and medium for accurately decoding Chinese real-time speech and instant punctuation. Background Art

[0002] In the field of automatic speech recognition (ASR) and text processing, correct punctuation is crucial to maintaining the semantics and structure of a sentence. Especially in real-time speech processing, the accurate placement of punctuation is even more important due to its unique grammatical and structural characteristics.

[0003] In the prior art, a predefined silence threshold is often used to determine the boundary of sentence segmentation. This will cause great problems in real-time processing of continuous speech and scenes where the speaker pauses irregularly or speaks very fast. It is impossible to accurately place punctuation marks in real-time speech processing, resulting in inaccurate information transmission and reduced readability of the text. This problem is particularly prominent for languages ​​with complex structures such as Chinese. For example: Voice Clip 1 I don't believe it. Voice Clip 2 I don't believe her. Voice Clip 3 I can't believe she would do such a thing! Audio Clip 4 I can't believe she would do such a thing, can you? As shown in the table above, users pause at unpredictable moments when thinking. Speech clips 1-4 in the table above are all possible, and each is a valid sentence with correct punctuation. If speech clip 4 is the user's intended sentence, then speech clips 1-3 are all incorrect. Summary of the invention

[0004] The purpose of the present invention is to provide a method for accurately decoding Chinese real-time speech punctuation to solve the technical problems existing in the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: A method for accurately decoding Chinese real-time speech punctuation, comprising the following steps: Step S1: training a punctuation prediction model; Step S2: receiving a converted text of Chinese real-time speech conversion, and dividing the converted text into a plurality of original texts according to a set time window; Step S3: pre-processing the original text: first, perform word segmentation, and then send it to the punctuation prediction model to perform punctuation prediction and output the text with correct punctuation; Step S4: outputting the text with punctuation marks in step S3 to a dynamic text stream receiver, wherein the dynamic text stream receiver includes a temporary storage area for storing sentence boundary text to be detected and a storage area for storing new input text; Step S5: Detect sentence boundaries: Analyze the text stored in the dynamic text stream receiver and search for sentence end markers. When a sentence end marker is detected, the sentence boundary is detected and the current sentence end marker is recorded; wherein the text includes the remaining text in the temporary storage area and the new input text in the storage area; Step S6: When the dynamic text stream receiver receives new text, it determines and outputs all previous complete sentences, where the complete sentence refers to the text between two sentence end tags; Step S7: Update the dynamic text stream receiver: put the remaining text into a temporary storage area; Step S8: Repeat steps S2 to S7 until all original text processing is completed.

[0006] Preferably, the method further comprises step S9: after executing step S8, if there is still text stored in the temporary storage area, the remaining text is output.

[0007] Preferably, the training method of the punctuation mark prediction model in step S1 is as follows: Step S1.1: Prepare Chinese training data, clean and filter it, and convert the original text with punctuation marks into a spoken text without punctuation marks; Step S1.2: Generate truth tags: Use the maintained alignment and the original text with punctuation to create truth punctuation tags and mark the spoken text; Word segmentation: Word segmentation of the original text with punctuation; Transformer encoding: Encode the words after word segmentation through the Transformer model; Punctuation Token classifier: Classify punctuation marks; Step S1.3: Based on the above settings, a punctuation prediction model based on bert chinese base is trained.

[0008] In step S3, the punctuation prediction model performs punctuation prediction in the following manner: Step S3.1: remove punctuation marks from the original text; Step S3.2: Create true punctuation labels and perform true value labeling on the original text with punctuation removed; Step S3.3: Token-level punctuation labeling: Add punctuation labels to each truth token; Step S3.4: Word-level punctuation tagging: Expand punctuation tags to the word level; Step S3.5: Generate final punctuation text: Combine all labels and processed text together to generate text with correct punctuation.

[0009] Preferably, in step S3.5, the label includes whether punctuation should be added to this position and the type of the punctuation.

[0010] Preferably, in step S3.5, the processing process of the processed text is as follows: if the two characters are directly O, no punctuation is added; if the two characters start with B, the punctuation type is determined according to the label.

[0011] Preferably, the sentence end mark is a punctuation mark representing the end of a sentence.

[0012] In order to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the method for accurate decoding of Chinese real-time speech punctuation.

[0013] In order to achieve the above-mentioned purpose, the present invention also provides a device for accurately decoding Chinese real-time speech punctuation, comprising: a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the device for accurately decoding Chinese real-time speech punctuation executes the method for accurately decoding Chinese real-time speech punctuation.

[0014] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention designs a dynamic text stream receiver based on a Chinese punctuation prediction model, which continuously adjusts to include text with undetermined sentence boundaries and newly input text in a temporary storage area, adds punctuation marks by dynamically detecting sentence boundaries, and outputs them immediately after determining the complete sentence, thereby improving the punctuation annotation accuracy of real-time text stream transcription and significantly improving the readability and semantic clarity of the text.

[0015] (2) The present invention is particularly suitable for real-time speech-to-text and online communication scenarios. It can make punctuation decisions across sentence boundaries, ensuring that the punctuation quality of the final text can be effectively improved regardless of the performance of the decoder segmentation mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the process of Example 1 of the present invention.

[0017] Figure 2 This is a principle block diagram of Example 3 of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to have a clearer understanding and knowledge of the present invention, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described below are only used to explain the present invention, and are convenient for understanding. The technical solutions provided by the present invention are not limited to the technical solutions provided by the following embodiments, and the technical solutions provided by the embodiments should not limit the protection scope of the present invention.

[0019] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0020] Example 1 like Figure 1 As shown, this embodiment provides a method for accurate decoding of punctuation in real-time Chinese speech. The method designs a dynamic text stream receiver based on a Chinese punctuation prediction model, which is continuously adjusted to include text with undetermined sentence boundaries and newly input text segments in a temporary storage area, adds punctuation marks by dynamically detecting sentence boundaries, and outputs them immediately after determining the complete sentence.

[0021] In this embodiment, the specific steps of the Chinese real-time speech instant punctuation accurate decoding method are as follows: 1. Step S1: training a punctuation prediction model; The punctuation prediction model can predict punctuation at the token level and word level. The training process is as follows: Step S1.1: Prepare Chinese training data, clean and filter it, and convert the original text with punctuation marks into a spoken text without punctuation marks; Step S1.2: Generate truth value tags: Use the original text with alignment and punctuation to create truth value punctuation tags and mark the spoken text; Word segmentation: The original text with punctuation is segmented. When segmenting, Jieba word segmentation can be used for Chinese. If other languages ​​are involved, BPE word segmentation can be used; Transformer encoding: Encode the segmented words through the Transformer model; Punctuation Token classifier: Classify punctuation marks; Step S1.3: Based on the above settings, a punctuation prediction model based on bert chinese base is trained.

[0022] Example of training process: The original text is Hello, what day is today? Step S1.1: Remove punctuation. How are you? What day is it today? Step S1.2: Generate truth value tags. O B_, O O O O B-? Among them, O, B_, and B-? are all truth value punctuation tags. The first O indicates that there is no punctuation between "you" and "hello", the second 'B_,' indicates that there is punctuation between "hello" and "today", and the punctuation is a comma, and so on.

[0023] The punctuation prediction model (or Chinese punctuation classifier) trained in this embodiment can deeply understand the Chinese context, combine more context, and show its unique advantages especially when understanding and processing languages with complex structures such as Chinese: understanding and processing the characteristics of complex languages such as Chinese; combined with a dynamic text stream receiver, accurately predicting punctuation within the dynamic text stream receiver.

[0024] II. Receive text Receive the converted text of real-time Chinese speech conversion, and divide the converted text into several original texts according to a set time window (the set time is a manually set value, for example: 2s).

[0025] III. Step S3: Predict punctuation Preprocess the original text: First, perform word segmentation, and after completion, send it into the punctuation prediction model for punctuation prediction and output the text with correct punctuation. The implementation method of punctuation prediction is as follows: Step S3.1: Remove the punctuation from the original text; Step S3.2: Create truth value punctuation tags and perform truth value marking on the original text after removing punctuation; Step S3.3: Token-level punctuation tags: Add punctuation tags to each truth value mark (each mark is actually a token) to indicate whether they are related to punctuation; Step S3.4: Word-level punctuation tags: Extend the punctuation tags to the word level (example "This is an example text.", after word segmentation, "text" is a token, the token-level tag is that there is a full stop after "text", and the word level is more precise, there is a full stop after the word "this"), to process punctuation more precisely; Step S3.5: Generate the final punctuation text: Combine all tags and the processed text together to generate the text with correct punctuation. Among them, the tags include whether punctuation should be added at this position and the category of punctuation. The processing process of the processed text is as follows: If it is O between two words, no punctuation is added. If it starts with B between two words, the punctuation type is judged according to the tag.

[0026] 4. Step S4: Dynamic decoding window processing The text with punctuation marks is output to a dynamic text stream receiver, which includes a temporary storage area for storing sentence boundary text to be detected and a storage area for storing newly input text. Initially, the window is empty, so the entire text is regarded as a newly input text segment; the dynamic text stream receiver stores the texts separately according to the order in which they are received. The text segments remaining after sentence boundary detection of the first received text are placed in the temporary storage area, and the newly input text is placed in the storage area.

[0027] 5. Step S5: Detecting sentence boundaries Analyze the text stored in the dynamic text stream receiver and look for the sentence end mark. When the sentence end mark is detected, the sentence boundary is detected and the current sentence end mark is recorded; wherein, the text includes the remaining text in the temporary storage area (i.e., the remaining text of the previous detection) and the new input text in the storage area, and the two parts are spliced ​​together to form the text currently detected; if the sentence boundary is not detected, that is, in this detection, it belongs to the remaining text, and the text is placed in the temporary storage area for the next detection. In this step, the sentence end mark is a punctuation mark representing the end of a sentence, such as a period, an exclamation mark, a question mark, etc.

[0028] 6. Step S6: Output the complete sentence When the dynamic text stream receiver receives new text, it determines and outputs all previous complete sentences, where a complete sentence refers to a sentence before a sentence end tag; sentences that are not output have no remaining text and are stored in a temporary storage area.

[0029] 7. Step S7: Update the dynamic text stream receiver The remaining text is placed in a temporary storage area to form the remaining text in the temporary storage area in the text for the next detection.

[0030] 8. Step S8: Repeat steps S2 to S7 until all original text processing is completed.

[0031] If there is still text stored in the temporary storage area after executing step S8, the remaining text is output.

[0032] Examples: Suppose there is a text: "This is a sample text. It has no punctuation, which makes it difficult to read. We need punctuation to clearly convey the meaning." Now, this text will be processed by the method provided in this embodiment: Input text segment: "This is a sample text. It has no punctuation, which makes it difficult to read. We need punctuation to clearly convey the meaning." Received text: "This is a sample text. It has no punctuation, which makes it difficult to read. We need punctuation to clearly convey the meaning." The input text is segmented and fed into the punctuation model for embedding and punctuation prediction. Dynamic decoding window: The text is put into the dynamic decoding window. Initially, the window is empty, so the entire text is treated as a newly input text segment. Detect sentence boundaries: Start analyzing the text and look for possible sentence end markers: Assume that the recognition of "text" and "difficult" should be followed by the end of the sentence. Emit complete sentences: Two sentence boundaries are found, and the following two sentences are emitted: · "This is a sample text." · "It has no punctuation, which makes it difficult to read." Update dynamic text receiver: The remaining text "We need punctuation to clearly convey the meaning." remains in the dynamic text receiver, waiting for the next piece of text to be input. Repeat processing: Since this is the last part of the text, the system will try to find the sentence boundary again, assuming that it recognizes the end of the sentence after "meaning". Finally, the system emits the remaining sentence: "We need punctuation to clearly convey the meaning." Example 2 This embodiment provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the Chinese real-time speech instant punctuation accurate decoding method provided in Example 1. A person of ordinary skill in the art can understand that all or part of the steps of implementing the method provided in Example 1 can be completed by hardware related to the computer program, and the above-mentioned computer program can be stored in a computer-readable storage medium, and when the program is executed, the steps of the method provided in Example 1 are executed; and the above-mentioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0033] Example 3 like Figure 2 As shown, this embodiment provides a Chinese real-time speech instant punctuation accurate decoding device, including: a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the Chinese real-time speech instant punctuation accurate decoding device executes the Chinese real-time speech instant punctuation accurate decoding method provided in Example 1.

[0034] Specifically, the memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0035] Preferably, the processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0036] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for accurately decoding Chinese real-time speech punctuation, characterized in that: The following steps are involved: Step S1: training a punctuation prediction model; Step S2: receiving a converted text of Chinese real-time speech conversion, and dividing the converted text into a plurality of original texts according to a set time window; Step S3: pre-processing the original text: first, perform word segmentation, and then send it to the punctuation prediction model to perform punctuation prediction and output the text with correct punctuation; Step S4: outputting the text with punctuation marks in step S3 to a dynamic text stream receiver, wherein the dynamic text stream receiver includes a temporary storage area for storing sentence boundary text to be detected and a storage area for storing new input text; Step S5: Detect sentence boundaries: Analyze the text stored in the dynamic text stream receiver and search for sentence end markers. When a sentence end marker is detected, the sentence boundary is detected and the current sentence end marker is recorded; wherein the text includes the remaining text in the temporary storage area and the new input text in the storage area; Step S6: When the dynamic text stream receiver receives new text, it determines and outputs all previous complete sentences, where the complete sentence refers to the sentence before a sentence end mark; Step S7: Update the dynamic text stream receiver: put the remaining text into a temporary storage area; Step S8: Repeat steps S2 to S7 until all original text processing is completed.

2. The Chinese real-time speech instant punctuation accurate decoding method according to claim 1 is characterized in that: The method further includes step S9: after executing step S8, if there is still text stored in the temporary storage area, the remaining text is output.

3. The Chinese real-time speech instant punctuation accurate decoding method according to claim 2 is characterized in that: The training method of the punctuation mark prediction model in step S1 is as follows: Step S1.1: Prepare Chinese training data, clean and filter it, and convert the original text with punctuation marks into a spoken text without punctuation marks; Step S1.2: Generate ground truth tags: Use the maintained alignment and the original text with punctuation to create ground truth punctuation tags and mark the spoken form text; Word segmentation: segment the original text with punctuation marks; Transformer encoding: encode the segmented words through the Transformer model; Punctuation Token classifier: classify punctuation marks; Step S1.3: Based on the above settings, a punctuation prediction model based on bert chinese base is trained.

4. The Chinese real-time speech instant punctuation accurate decoding method according to claim 3 is characterized in that: In step S3, the punctuation prediction model performs punctuation prediction in the following manner: Step S3.1: remove punctuation marks from the original text; Step S3.2: Create true punctuation labels and perform true value labeling on the original text with punctuation removed; Step S3.3: Token-level punctuation labeling: Add punctuation labels to each truth token; Step S3.4: Word-level punctuation tagging: Expand punctuation tags to the word level; Step S3.5: Generate final punctuation text: Combine all labels and processed text together to generate text with correct punctuation.

5. The Chinese real-time speech instant punctuation accurate decoding method according to claim 4 is characterized in that: In step S3.5, the label includes whether punctuation should be added to this position and the type of the punctuation.

6. The Chinese real-time speech instant punctuation accurate decoding method according to claim 5 is characterized in that: In step S3.5, the processing process of the processed text is as follows: if the two characters are directly O, no punctuation is added; if the two characters start with B, the punctuation type is determined according to the label.

7. The Chinese real-time speech instant punctuation accurate decoding method according to claim 6 is characterized in that: The sentence end mark is a punctuation mark representing the end of a sentence.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the Chinese real-time speech instant punctuation accurate decoding method as described in any one of claims 1 to 7.

9. A Chinese real-time speech instant punctuation accurate decoding device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory, so that the Chinese real-time speech instant punctuation accurate decoding device executes the Chinese real-time speech instant punctuation accurate decoding method as described in any one of claims 1 to 7.

Images