An audio recognition method based on an acoustic model and a language model

Abstract

This application provides an audio recognition method based on acoustic and language models. The method includes: acquiring an audio segment; processing the audio segment using an acoustic model to obtain a phoneme sequence corresponding to the audio segment, the phoneme sequence including punctuation phonemes; and processing the phoneme sequence using a language model to obtain text information corresponding to the phoneme sequence, the text information including punctuation marks. This method eliminates the need for additional punctuation and search modules to process plain text information; it directly obtains the text information with punctuation marks corresponding to the audio segment through acoustic and language models, making audio recognition more convenient and efficient.

Description

Technical Field

[0001] This application relates to the field of audio signal processing, and more specifically, to an audio recognition method based on acoustic and language models. Background Technology

[0002] In existing audio recognition technologies, audio recognition models can only convert audio segments into plain text information, that is, text information excluding punctuation marks.

[0003] Furthermore, if punctuation marks need to be added to plain text information, a punctuation module and a punctuation path search module are required to process the plain text information. Therefore, in existing technologies, it is not possible to directly obtain text information with punctuation marks using a single audio recognition model. Summary of the Invention

[0004] This application provides an audio recognition method based on acoustic and language models.

[0005] In a first aspect, some embodiments of this application provide an audio recognition method based on an acoustic model and a language model. The method includes: acquiring an audio segment; processing the audio segment based on the acoustic model to obtain a phoneme sequence corresponding to the audio segment; wherein the phoneme sequence includes punctuation phonemes for representing punctuation marks; the acoustic model is obtained by training an initial acoustic model using a first training audio segment; the first training audio segment is labeled with a first training phoneme sequence, and the first training phoneme sequence includes first training punctuation phonemes; and processing the phoneme sequence based on the language model to obtain text information corresponding to the phoneme sequence; wherein the text information includes punctuation marks; the language model is obtained by training an initial language model using a second training phoneme sequence; the second training phoneme sequence is labeled with first training text information; the second training phoneme sequence includes second training punctuation phonemes; and the first training text information includes training punctuation marks.

[0006] This application provides an audio recognition method based on acoustic and language models. The method processes audio segments using an acoustic model to obtain a phoneme sequence containing punctuation phonemes, where punctuation phonemes are phonemes representing punctuation marks. The language model then further processes this phoneme sequence to obtain text information containing punctuation marks. This method eliminates the need for additional punctuation and search modules to process plain text information; it directly obtains the text information containing punctuation marks corresponding to the audio segment through the acoustic and language models, making audio recognition more convenient and efficient. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This illustration shows an application environment diagram of an audio recognition method based on acoustic and language models provided in an embodiment of this application.

[0009] Figure 2 The diagram shows a flowchart of an audio recognition method based on an acoustic model and a language model provided in the first embodiment of this application.

[0010] Figure 3 The diagram shows a flowchart of an audio recognition method based on an acoustic model and a language model provided in the second embodiment of this application.

[0011] Figure 4 The diagram shows a flowchart of an audio recognition method based on an acoustic model and a language model provided in the third embodiment of this application.

[0012] Figure 5 The diagram shows a flowchart of an audio recognition method based on an acoustic model and a language model provided in the fourth embodiment of this application.

[0013] Figure 6 The diagram shows a flowchart of a training method for an acoustic model provided in an embodiment of this application.

[0014] Figure 7 The diagram shows a flowchart of a language model training method provided in an embodiment of this application.

[0015] Figure 8 This illustration shows a block diagram of an audio recognition device based on an acoustic model and a language model, according to an embodiment of this application.

[0016] Figure 9 A block diagram of a smart device provided in an embodiment of this application is shown.

[0017] Figure 10 A block diagram of a computer-readable storage medium provided in an embodiment of this application is shown. Detailed Implementation

[0018] The embodiments of this application are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0019] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0020] This application provides an audio recognition method based on acoustic and language models. The method processes audio segments using an acoustic model to obtain a phoneme sequence containing punctuation phonemes, where punctuation phonemes are phonemes representing punctuation marks. The language model then further processes this phoneme sequence to obtain text information containing punctuation marks. This method eliminates the need for additional punctuation and search modules to process plain text information; it directly obtains the text information containing punctuation marks corresponding to the audio segment through the acoustic and language models, making audio recognition more convenient and efficient.

[0021] To facilitate a detailed explanation of the present application, the application environment of the embodiments of the present application will be described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 An electronic device 100 is provided in this application embodiment, which can execute the audio recognition method based on acoustic model and language model provided in this application embodiment.

[0022] The electronic device 100 can be a smartphone, tablet computer, personal computer, etc. In this embodiment, the electronic device 100 has an audio recognition function. Optionally, the electronic device 100 pre-stores an acoustic model and a language model. The audio segment is processed by the acoustic model to obtain a phoneme sequence, and the phoneme sequence is processed by the language model to obtain the corresponding text information with punctuation marks.

[0023] The aforementioned acoustic model and language model can be trained by the electronic device 100. In some embodiments, the electronic device 100 trains the initial acoustic model based on a first training audio segment to obtain the aforementioned acoustic model, and trains the initial language model based on a second training phoneme sequence to obtain the aforementioned language model. In other embodiments, the acoustic model and language model are trained in a server 110, and the server 110 sends the trained acoustic model and language model to the electronic device 100.

[0024] In some embodiments, the electronic device 100 also has an audio acquisition function, which is used to acquire audio segments to be identified as text information. Optionally, the electronic device 100 implements the audio acquisition function through its own microphone assembly, or a microphone assembly with which it has established a communication connection.

[0025] In some embodiments, the electronic device 100 has a designated application installed, and the audio recognition function is implemented through the designated application. The designated application refers to an application where audio recognition is required, such as a social application, a recording application, or a conferencing application. In this embodiment, the execution subject of each step can be an electronic device or the designated application. In this embodiment, the description only uses an electronic device as the execution subject of each step.

[0026] Please see Figure 2 , Figure 2 This illustration schematically depicts an audio recognition method based on an acoustic model and a language model provided in the first embodiment of this application. In this method, the text information corresponding to an audio segment, including punctuation marks, can be directly obtained through the acoustic model and the language model. Specifically, this method includes steps S210 to S230.

[0027] Step S210: Obtain the audio segment.

[0028] An audio clip refers to a carrier of information about the frequency and amplitude variations of sound waves that contain speech, music, or sound effects. Audio clips can originate from report audio in a meeting setting, call audio in a telephone setting, or spoken audio in everyday communication scenarios, etc.

[0029] In some embodiments, the electronic device captures audio segments in real time via a microphone. For example, the audio segment might be a report audio from a meeting. When the microphone enters recording mode, it picks up the report audio from the meeting to acquire the audio segment.

[0030] In some other embodiments, the electronic device reads an audio segment from a local storage location. Exemplarily, a social application is installed in the electronic device (e.g., a smart phone), and the application receives audio segments sent by other contacts and stores the audio segments in a specified storage path set by the application. The electronic device reads the audio segment from the above-specified storage path.

[0031] The electronic device obtains an audio segment after receiving an audio recognition instruction. In some embodiments, in an online meeting scenario, the electronic device displays a meeting interface, and the meeting interface includes a voice recognition control. If a trigger signal for the voice recognition control is received, the electronic device obtains an audio recognition instruction and obtains the currently playing audio segment as the audio segment to be recognized into text information. In some other embodiments, in a social conversation scenario, the electronic device displays a conversation interface, and the conversation interface includes voice messages sent by other contacts. After the user long-presses the voice message, a menu bar is superimposed and displayed on the conversation interface. The menu bar includes a voice-to-text control. If a trigger signal for the voice-to-text control is received, the electronic device obtains an audio recognition instruction and obtains the above voice message as the audio segment to be recognized into text information.

[0032] Step S220: Process the audio segment based on an acoustic model to obtain a phoneme sequence corresponding to the audio segment, and the phoneme sequence includes punctuation phonemes for representing punctuation marks.

[0033] An acoustic model refers to a model that extracts the speech features of an audio segment and converts the speech features into a phoneme sequence. In the method proposed in this application, the acoustic model is obtained by training an initial acoustic model with a first training audio segment. The first training audio segment is labeled with a first training phoneme sequence, and the first training phoneme sequence includes first training punctuation phonemes. The process of training the initial acoustic model with the first training audio segment is introduced in the following embodiments.

[0034] A phoneme sequence is a sequence composed of at least one phoneme, and a phoneme is the smallest speech unit divided according to the natural attributes of speech. Taking Chinese phonemes as an example, the phonemes corresponding to the three characters "Putonghua" are "p,u,t,o,ng,h,u,a". In this application, the acoustic model can convert the audio segment corresponding to the three characters "Putonghua" in the audio segment into the phoneme sequence "p,u,t,o,ng,h,u,a".

[0035] In this embodiment, the electronic device uses the audio segment as the input of the acoustic model, and the output of the acoustic model is a phoneme sequence. Among them, the phoneme sequence includes punctuation phonemes used to represent punctuation marks. As an implementation manner, the punctuation phoneme can be represented by a silent sound, and the silent sound can be "sil". As another implementation manner, the punctuation phoneme can also be represented by an unknown sound, and the unknown sound can be "unk". Exemplarily, taking the audio segment "China () good" as an example, where "()" is a pause segment, the acoustic model uses the audio segment corresponding to "China () good" as the model input, and the corresponding model output is "zh,o,ng,g,u,o,sil,h,a,o", or "zh,o,ng,g,u,o,unk,h,a,o". Through the above processing method, when the subsequent speech model processes the phoneme sequence, it can determine the corresponding punctuation mark based on the punctuation phoneme, and finally output the text information with punctuation marks.

[0036] There are usually audio sub-segments in the audio segment from which speech features cannot be extracted, such as noise segments, pause segments, etc. in the audio segment. In some embodiments, for the audio sub-segments in the audio segment from which speech features cannot be extracted, if the average loudness value of the audio sub-segment is greater than the preset loudness value, it is determined as a noise segment and represented by an unknown sound. When the noise component in the audio sub-segment is large, the acoustic model cannot extract speech features from the audio sub-segment. In this case, the acoustic model will output an unknown sound to represent the noise segment. In other embodiments, if the average loudness value of the audio sub-segment is less than or equal to the preset loudness value, it is determined as a pause segment and represented by a silent sound. Generally, speakers have the following language habits: there is a pause between two semantically complete sentences. That is, there is a certain pause time between two semantically complete and adjacent sentences, and the audio segment corresponding to the above pause time is the pause segment. When the acoustic model processes the audio segment, when it is determined that there is a pause segment in the current audio segment, since the corresponding speech features cannot be obtained from the pause segment, in this case, the acoustic model will output a silent sound to represent the pause segment.

[0037] It should be noted that in the audio recognition method based on the acoustic model and the language model proposed in this application, there can be multiple different acoustic models, and the multiple different acoustic models are used to process audio segments of different languages (for example, Chinese, English, French, etc.). When the electronic device determines the language corresponding to the current audio segment, it further determines the acoustic model corresponding to the language.

[0038] Step S230: Process the phoneme sequence based on the language model to obtain the text information corresponding to the phoneme sequence, and the text information includes punctuation marks.

[0039] A language model refers to a model that converts a sequence of phonemes into text information. In the method proposed in this application, the language model is obtained by training an initial language model with a second training phoneme sequence. The second training phoneme sequence is labeled with first training text information, the second training phoneme sequence includes second training punctuation phonemes, and the first training text information includes training punctuation marks. The process of training the initial language model with the second training phoneme sequence will be introduced in the following embodiments.

[0040] In this embodiment, the model input of the language model is a phoneme sequence including punctuation phonemes, and the model output of the language model is text information with punctuation marks. Exemplarily, taking the phoneme sequence including punctuation phonemes "zh,o,ng,g,u,o,sil,h,a,o" as an example, the language model takes "zh,o,ng,g,u,o,sil,h,a,o" as the model input, and the corresponding model output is "China, good".

[0041] This application provides an audio recognition method based on an acoustic model and a language model. This method processes an audio segment through the acoustic model to obtain a phoneme sequence with punctuation phonemes, where the punctuation phoneme is a phoneme representing a punctuation mark. Further, the phoneme sequence with punctuation phonemes is processed through the language model to obtain text information with punctuation marks. This method does not need to introduce other punctuation modules and search modules to process pure text information, and can directly obtain the text information with punctuation marks corresponding to the audio segment through the acoustic model and the language model, making audio recognition more convenient and efficient.

[0042] Please refer to Figure 3 , Figure 3 which schematically shows an audio recognition method provided in the second embodiment of this application. In this method, the audio segment includes at least one pause segment, and the position of the punctuation phoneme in the phoneme sequence can be determined based on the position information of the pause segment; further, the text information corresponding to the phoneme sequence is determined based on the candidate probabilities of at least one candidate text. Specifically, this method includes steps S310 to step S360.

[0043] Step S310: Obtain an audio segment.

[0044] The specific implementation manner of step S310 can refer to the detailed description in step S210, and will not be elaborated here one by one.

[0045] Step S320: Process the audio segment based on the acoustic model to determine at least one pause segment.

[0046] In this embodiment, the audio segment includes at least one pause segment, which represents a segment with a loudness value less than a preset loudness value and a duration greater than or equal to a preset duration. The preset loudness value can be a default value in the acoustic model, or it can be dynamically adjusted by researchers based on actual model training data. For example, the preset loudness value can be any value less than 10 dB. The preset duration can be a default value in the acoustic model, or it can be dynamically adjusted by researchers based on actual model training data. For example, the preset duration can be any value greater than 0.1 s.

[0047] Specifically, step S320 may include steps S321 to S325.

[0048] Step S321: Based on the acoustic model, obtain the loudness values ​​of the audio segment at different times.

[0049] As one implementation method, the acoustic model includes a preprocessing module, which obtains the loudness values ​​of the audio segment at different times.

[0050] Step S323: Determine the audio sub-segments with loudness values ​​less than the preset loudness value as candidate pause segments.

[0051] The preprocessing module identifies audio segments with an average loudness value less than a preset loudness value as candidate pause segments. For example, with a preset loudness value of 1dB, the preprocessing module identifies audio segments with an average loudness value less than 1dB as candidate pause segments.

[0052] Step S325: If the duration of the candidate pause segment is greater than or equal to the preset duration, then the candidate pause segment is determined as a pause segment.

[0053] The preprocessing module calculates the duration of candidate pause segments and determines whether a candidate pause segment is indeed a pause segment based on its duration. If the duration of a candidate pause segment is greater than or equal to a preset duration, it is identified as a pause segment; otherwise, if the audio duration of a candidate pause segment is less than the preset duration, it is determined not to be a pause segment. For example, with a preset duration of 1 second, if the duration of a candidate pause segment is greater than or equal to 1 second, it is identified as a pause segment.

[0054] Step S330: Determine the location information of at least one pause segment based on at least one pause segment.

[0055] Location information represents the position of a pause segment within an audio segment. As one implementation, the location information of a pause segment can be determined based on at least two of the following: the start time, the end time, and the duration of the pause segment.

[0056] Step S340: Based on the location information, obtain the phoneme sequence corresponding to the audio segment. The phoneme sequence includes at least one punctuation phoneme for representing punctuation marks.

[0057] The punctuation phonemes include the silence phoneme and / or the unknown phoneme. For the specific introduction of the silence phoneme and the unknown phoneme, reference can be made to the specific introduction in step S220.

[0058] The electronic device determines the position of the punctuation phoneme in the phoneme sequence based on the position information of the pause segment in the audio segment. Taking the audio segment "China () good" as an example, the pause segment "()" is between the audio segment "China" and the audio segment "good". The punctuation phoneme corresponding to this pause segment "()" is the silence phoneme "sil" or the unknown phoneme "unk". The electronic device first obtains the phoneme sequences corresponding to the audio segments "China" and "good" outside the pause segment, which are "zh,o,ng,g,u,o" and "h,a,o" respectively. Since the pause segment "()" is between the audio segment "China" and the audio segment "good", it can be determined that the silence phoneme "sil" or the unknown phoneme "unk" is between the phoneme sequence "zh,o,ng,g,u,o" and the phoneme sequence "h,a,o". Therefore, the phoneme sequence corresponding to the audio segment "China () good" is "zh,o,ng,g,u,o,sil,h,a,o", or "zh,o,ng,g,u,o,unk,h,a,o".

[0059] In the embodiment of the present application, determining the position of the punctuation phoneme in the phoneme sequence through the position information of the pause segment in the audio segment helps to quickly locate the punctuation phoneme in the subsequent phoneme sequence.

[0060] Step S350: Process the phoneme sequence based on the language model to obtain the candidate probabilities of at least one candidate text.

[0061] The candidate text includes at least one candidate character and at least one candidate punctuation mark. The candidate probability represents the probability that the candidate text is the text information corresponding to the phoneme sequence.

[0062] The language model first determines at least one candidate text through the dictionary module. Specifically, the language model determines the candidate characters corresponding to the non-punctuation phonemes in the phoneme sequence by looking up the mapping relationship between phonemes and candidate characters in the dictionary module. The dictionary module can be set inside the language model or be independent outside the language model. The dictionary module represents the mapping relationship between phonemes and candidate characters, and this mapping relationship can be a one-to-one mapping relationship or a one-to-many mapping relationship. Exemplarily, based on the dictionary module, the candidate characters corresponding to the phoneme sequence "zh,o,ng,g,u,o" can be "中国" (China), or can also be "种过" (planted a country). And based on the dictionary module, the candidate characters corresponding to the phoneme sequence "h,a,o" can be "好" (good), or can also be "豪" (luxurious). Specifically, the dictionary module can be Chinese dictionaries such as thchs30, CC-CEDICT, etc., and English dictionaries such as CMU-dict, etc.

[0063] In the embodiments of the present application, the language model can also determine at least one candidate punctuation mark through the dictionary module. In some embodiments, the dictionary module includes a punctuation mark dictionary sub-module, and the punctuation mark sub-dictionary module represents the mapping relationship between punctuation phonemes and punctuation marks. Exemplarily, the mapping relationship between punctuation phonemes and punctuation marks is shown in Table-1.

[0064] Table-1

[0065]

[0066] Specifically, the language model determines the candidate punctuation marks corresponding to the punctuation phonemes in the phoneme sequence by looking up the mapping relationship between punctuation phonemes and punctuation marks in the punctuation mark dictionary sub-module. Exemplarily, based on the punctuation mark sub-dictionary module, the candidate punctuation mark corresponding to "sil" can be "," (comma), or can also be "。" (period).

[0067] In some embodiments, the electronic device calculates the candidate probability of the candidate text based on the character candidate probability of at least one candidate character and the punctuation mark candidate probability of at least one candidate punctuation mark.

[0068] Optionally, step S350 includes steps S3501 to S3503.

[0069] Step S3501: Process the phoneme sequence based on the language model to obtain at least one character candidate probability corresponding to at least one candidate text.

[0070] After the language model determines at least one candidate character and at least one candidate punctuation mark, it respectively determines the character candidate probability corresponding to the candidate character and the punctuation mark candidate probability corresponding to the candidate punctuation mark. In this embodiment, the character candidate probability and the punctuation mark candidate probability are determined by the language model based on the context features of the audio segment. The context features of the audio segment include, but are not limited to, the词性 of the vocabulary (e.g., noun, verb, adjective, etc.), the词频 of the vocabulary, and the integrity of the sentence. Among them, the词频 represents the number of times the vocabulary appears.

[0071] Step S3502: Process the phoneme sequence based on the language model to obtain at least one punctuation mark candidate probability corresponding to at least one candidate text.

[0072] After the language model determines at least one candidate punctuation mark, it determines the punctuation mark candidate probability corresponding to the candidate punctuation mark. The specific implementation manner of determining the punctuation mark candidate probability refers to the introduction in step S3501, and will not be elaborated here one by one.

[0073] Step S3503: Determine the candidate probability of at least one candidate text as the product of at least one character candidate probability and at least one punctuation mark candidate probability.

[0074] The electronic device determines the candidate probability of at least one candidate text as the product of at least one character candidate probability and at least one punctuation mark candidate probability.

[0075] Exemplarily, the character candidate probability corresponding to "中国" is 90%, and the character candidate probability corresponding to "种过" is 10%. The character candidate probability corresponding to "好" is 90%, and the character candidate probability corresponding to "豪" is 10%. The punctuation mark candidate probability corresponding to "," is "85%", and the punctuation mark candidate probability corresponding to "。" is "15%".

[0076] Taking the candidate text "中国,好" as an example, the candidate probability corresponding to "中国,好" is 90% * 85% * 90% = 68.85%. Similarly, the candidate probability corresponding to the candidate text "种过。好" is 1.35% and the candidate probability corresponding to "中国,豪" is 7.65%.

[0077] In some embodiments, when the punctuation mark candidate probability is greater than or equal to the existence probability threshold, the electronic device determines that there is a corresponding punctuation mark in the candidate text. When the punctuation mark candidate probability is less than the existence probability threshold, it determines that there is no corresponding punctuation mark in the candidate text. The existence probability threshold can be the default value in the language model or can be dynamically adjusted by researchers based on the actual model training data. Exemplarily, the existence probability threshold can be any value greater than 50% and less than 100%.

[0078] Step S360: Determine the candidate text corresponding to the maximum candidate probability as the text information corresponding to the phoneme sequence.

[0079] In this embodiment, the candidate text corresponding to the maximum candidate probability among multiple candidate probabilities is determined as the text information of the phoneme sequence. As an implementation manner, the maximum candidate probability can be determined by a sorting algorithm. Specifically, the multiple candidate probabilities corresponding to multiple candidate texts are sorted by a sorting algorithm to determine the maximum candidate probability among the multiple candidate probabilities. The sorting algorithm includes but is not limited to bubble sort, selection sort, insertion sort, Shell sort, merge sort, etc. Exemplarily, based on the sorting algorithm, it can be determined that the maximum candidate probability among the above candidate probabilities is 68.85%. In the case of determining the maximum candidate probability, the candidate text "China, good" corresponding to the maximum candidate probability is determined as the text information corresponding to the phoneme sequence.

[0080] This application provides an audio recognition method based on an acoustic model and a language model. This method determines the position information of pause segments in an audio segment through the acoustic model, and further determines a phoneme sequence including punctuation phonemes in the case of determining the position information; further obtains the candidate probabilities of at least one candidate text through the language model, and determines the candidate text corresponding to the maximum candidate probability as the text information corresponding to the phoneme sequence. This method does not need to introduce other punctuation modules and search modules to process pure text information, and can directly obtain the text information with punctuation marks corresponding to the audio segment through the acoustic model and the language model, making audio recognition more convenient and efficient.

[0081] Please refer to Figure 4 , Figure 4 which schematically shows an audio recognition method based on an acoustic model and a language model provided in the third embodiment of this application. In this method, when obtaining the word candidate probability corresponding to the candidate text through the speech model, it will further determine whether the target word in the candidate text exists in the hot word library. If it exists in the hot word library, the corresponding word candidate probability is updated to improve the candidate probability of the candidate text containing the hot word, thereby improving the accuracy of audio recognition. Specifically, this method includes Step S410 to Step S480.

[0082] Step S410: Obtain an audio segment.

[0083] Step S420: Process the audio segment based on the acoustic model to obtain a phoneme sequence corresponding to the audio segment, and the phoneme sequence includes punctuation phonemes for representing punctuation marks.

[0084] Step S430: Process the phoneme sequence based on the language model to obtain at least one word candidate probability corresponding to at least one candidate text.

[0085] The specific implementation methods of steps S410 to S430 can be found in the detailed descriptions of steps S210 to S220 and step S3501, and will not be repeated here.

[0086] Step S440: Detect whether the target word in the candidate text exists in the hot word library.

[0087] The hot word lexicon is a lexicon consisting of at least one hot word. Hot words represent technical terms in a professional field, such as Markov models, Hilbert transforms, etc. Because hot words are used less frequently in everyday life, when the language model obtains at least one candidate text containing a hot word, the probability of the candidate text corresponding to that hot word is lower than the probability of the candidate text corresponding to other candidate texts.

[0088] The hot word library can be the default library in the language model, or it can be dynamically adjusted by researchers based on the test results of the actual language model. For example, when testing the language model, if the number of incorrect recognitions of a hot word exceeds a first preset number, the corresponding hot word will be added to the hot word library. Specifically, if the language model incorrectly identifies "Markov model" as "horse customer service model" and the number of such errors exceeds the first preset number, then "Markov model" will be added to the hot word library. The first preset number can be a default value in the language model, or it can be dynamically adjusted by researchers based on the test results of the actual language model.

[0089] The target word is any word in the candidate text. The electronic device compares each word in the candidate text with each hot word in the hot word database to determine whether the target word exists in the hot word database. Optionally, the electronic device uses a text comparison algorithm to determine whether the target word in the candidate text exists in the hot word database. Specifically, the text comparison algorithm can be an edit distance algorithm, an algorithm based on the longest common substring, etc.

[0090] Step S450: If the target word in the candidate text exists in the hot word library, then update the text candidate probability corresponding to the target word.

[0091] The updated probability of the target word's corresponding text candidate is greater than the probability of the target word's corresponding text candidate before the update. As one method, the probability mapping operation can be used to update the probability of the target word's corresponding text candidate. In some embodiments, the probability mapping operation is a linear mapping operation; in other embodiments, the probability mapping operation is a non-linear mapping operation. This application embodiment provides an implementation where the probability mapping operation is a linear mapping operation. Specifically, step S450 may include step S4501.

[0092] Step S4501: If the target word in the candidate text exists in the hot word library, then the product between the text candidate probability corresponding to the target word and the first mapping factor is determined as the updated text candidate probability corresponding to the target word.

[0093] The first mapping factor can be a default value in the language model, or it can be dynamically adjusted by researchers based on the test results of the actual language model. Specifically, the first mapping factor is greater than one. Taking a first mapping factor of 5 as an example, if the probability of the candidate text corresponding to the target word is 15%, then the probability of the candidate text corresponding to the updated target word will be 75%.

[0094] Step S460: Process the phoneme sequence based on the language model to obtain the candidate probability of at least one punctuation mark corresponding to at least one candidate text.

[0095] Step S470: The product of at least one candidate text probability and at least one candidate punctuation mark probability is determined as the candidate probability of at least one candidate text. The at least one candidate text probability includes the candidate text probability corresponding to the updated target word.

[0096] Step S480: Determine the candidate text corresponding to the highest candidate probability as the text information corresponding to the phoneme sequence.

[0097] The specific implementation methods of steps S460 to S480 can be found in the detailed descriptions of steps S3502 to S360, and will not be repeated here.

[0098] This application provides an audio recognition method based on acoustic and language models. In this method, when obtaining the text candidate probability corresponding to a candidate text through a speech model, it further determines whether the target word in the candidate text exists in a hot word database. If it exists, the corresponding text candidate probability is updated. Based on the updated text candidate probability, new text information corresponding to the phoneme sequence can be further determined. This method optimizes the text information output by the language model using a hot word database, resulting in higher audio recognition accuracy for the optimized text information.

[0099] Please see Figure 5 , Figure 5 This illustration schematically demonstrates an audio recognition method based on an acoustic model and a language model provided in the fourth embodiment of this application. In this method, when obtaining the text candidate probability corresponding to the candidate text through the speech model, the first word in the candidate text is further replaced with the second word based on the mapping table, thereby further updating the text candidate probability corresponding to the candidate text. Specifically, this method includes steps S510 to S580.

[0100] Step S510: Obtain the audio segment.

[0101] Step S520: Process the audio segment based on the acoustic model to obtain the phoneme sequence corresponding to the audio segment. The phoneme sequence includes punctuation phonemes used to represent punctuation marks.

[0102] Step S530: Process the phoneme sequence based on the language model to obtain the probability of at least one candidate text corresponding to at least one candidate text.

[0103] The specific implementation methods of steps S510 to S530 can be found in the detailed descriptions of steps S210 to S220 and step S3501, and will not be repeated here.

[0104] Step S540: Replace the first word in the candidate text with the second word based on the preset mapping table.

[0105] The preset mapping table includes at least one incorrect identification result and the correct identification result corresponding to the incorrect identification result; the first word is the incorrect identification result in the preset mapping table, and the second word is the correct identification result in the preset mapping table.

[0106] A preset mapping table represents the mapping relationship between incorrect and correct recognition results. In some embodiments, due to certain errors in the language model, some words cannot be correctly recognized. Therefore, during the testing of the language model, researchers establish a corresponding preset mapping table based on the model's test results. For example, when testing the language model, if the number of incorrect recognition results exceeds a second preset number, the corresponding correct recognition result is determined. A mapping relationship is established between this incorrect recognition result and the corresponding correct recognition result, and this mapping relationship is stored in the preset mapping table. Specifically, when testing the language model, if the language model incorrectly recognizes "Markov model" as "Horse Customer Service Model," then "Horse Customer Service Model" is the incorrect recognition result. If the number of incorrect recognitions exceeds the second preset number, the corresponding correct recognition result "Markov model" is determined. A mapping relationship is established between "Horse Customer Service Model" and "Markov Model," and this mapping relationship is stored in the preset mapping table. The second preset number can be a default value in the language model, or it can be dynamically adjusted by researchers based on the actual test results of the language model.

[0107] As one implementation method, a text comparison algorithm can be used to determine whether a first word exists in the candidate text, and if the first word exists in the candidate text, the first word can be replaced with a second word.

[0108] Step S550: Update the probability of the candidate text corresponding to the first word.

[0109] The updated candidate text probability of the first word is greater than the candidate text probability of the first word before the update. The updated candidate text probability is used to determine the candidate probability of the candidate text.

[0110] One approach is to update the text candidate probabilities using a probabilistic mapping operation. In some embodiments, the probabilistic mapping operation is a linear mapping operation. Specifically, the updated text candidate probability is determined by multiplying the original text candidate probability by a second mapping factor. The second mapping factor can be a default value in the language model or can be dynamically adjusted by researchers based on test results of the actual language model. Specifically, the second mapping factor is greater than one. For example, if the original text candidate probability is 15% with a second mapping factor of 5, the updated text candidate probability will be 75%.

[0111] Step S560: Process the phoneme sequence based on the language model to obtain the candidate probability of at least one punctuation mark corresponding to at least one candidate text.

[0112] Step S570: The product of the candidate probability of at least one text and the candidate probability of at least one punctuation mark is determined as the candidate probability of at least one candidate text.

[0113] At least one text candidate probability includes the updated text candidate probability corresponding to the first word.

[0114] Step S580: Determine the candidate text corresponding to the highest candidate probability as the text information corresponding to the phoneme sequence.

[0115] The specific implementation methods of steps S560 to S580 can be found in the detailed descriptions of steps S3502 to S360, and will not be repeated here.

[0116] This application provides an audio recognition method based on acoustic and language models. In this method, when obtaining the text candidate probabilities corresponding to candidate text through a speech model, the first word in the candidate text is further replaced with the second word based on a mapping table, thus updating the text candidate probabilities. Based on the updated text candidate probabilities, new text information corresponding to the phoneme sequence can be further determined. This method optimizes the text information output by the language model through a mapping table, resulting in higher audio recognition accuracy for the optimized text information.

[0117] Please see Figure 6 , Figure 6 The illustration schematically depicts a method for training an acoustic model according to an embodiment of this application. The method includes steps S610 to S640.

[0118] Step S610: Obtain the predicted phoneme sequence corresponding to the first training audio segment through the initial acoustic model.

[0119] The initial acoustic model is an acoustic model with incomplete parameter training. As one implementation method, the initial acoustic model can be a Hidden Markov Model (HMM), a Chain States model, etc., and is not specifically limited in this application. The first training audio segment is the training data for the initial acoustic model. The first training audio segment is labeled with a first training phoneme sequence, which includes first training punctuation phonemes. For example, taking the first training audio segment as “model() as”, where “()” represents a pause segment, the first training phoneme sequence corresponding to this first training audio segment is “m,o,x,i,ng,sil,z,u,o,w,e,i”. Using the initial acoustic model, the first training audio segment can be converted into a predicted phoneme sequence.

[0120] Step S620: Obtain the first error information between the predicted phoneme sequence and the first training phoneme sequence.

[0121] The processor compares the predicted phoneme sequence with the first training phoneme sequence to obtain the first error information between them. As one implementation, the predicted phoneme sequence can be converted into a first feature matrix, and the first training phoneme sequence can be converted into a second feature matrix. The first error information is obtained by calculating the difference between the first and second feature matrices. Optionally, the first error information can also be determined using methods such as maximum likelihood estimation, cross-entropy loss function, and maximum mutual information estimation.

[0122] Step S630: If the first error information does not meet the first preset condition, then the parameters of the initial acoustic model are optimized based on the first error information and the first optimization function, and the step of obtaining the predicted phoneme sequence corresponding to the first training audio segment through the initial acoustic model is restarted.

[0123] The first preset condition characterizes whether parameter optimization training is complete. If the first error information does not meet the first preset condition, it indicates that the error between the predicted phoneme sequence obtained through the initial acoustic model and the first training phoneme sequence is large, requiring optimization and adjustment of the parameters in the initial acoustic model. In some embodiments, the first preset condition refers to the first error information between the predicted phoneme sequence and the first training phoneme sequence being less than a first preset value. The aforementioned first preset value is actually determined according to the accuracy requirements of the acoustic model. If the accuracy requirements of the acoustic model are high, the first preset value is small; if the accuracy requirements of the acoustic model are low, the first preset value is large.

[0124] The first optimization function represents a function that optimizes and adjusts the model parameters of the initial acoustic model. The first optimization function includes, but is not limited to, batch gradient descent, stochastic gradient descent, and adaptive optimization functions. Taking the first error information as input, the first optimization function optimizes and adjusts the parameters in the initial acoustic model so that the predicted phoneme sequence output by the optimized initial acoustic model is closer to the first training phoneme sequence.

[0125] Furthermore, the optimized and adjusted initial acoustic model replaces the initial acoustic model in step S610, and the execution restarts from step S610.

[0126] Step S640: If the first error information meets the first preset condition, then obtain the initial acoustic model that has completed training.

[0127] If the first error information meets the first preset condition, it means that the first error information between the predicted phoneme sequence obtained by the initial acoustic model and the first training phoneme sequence is small. At this time, the initial acoustic model completes the model optimization training.

[0128] This application provides a method for training an acoustic model, which can optimize the training of the acoustic model, resulting in a higher accuracy in acquiring phoneme sequences after training.

[0129] Please see Figure 7 , Figure 7 The illustration schematically depicts a language model training method provided in an embodiment of this application. The method includes steps S710 to S740.

[0130] Step S710: Obtain the first predicted text information corresponding to the second training phoneme sequence through the initial language model.

[0131] The initial language model is trained as a language model with incomplete parameter training. As an implementation, the initial language model can be a probabilistic grammar rule model, a Chinese language model (N-gram), a neural network model (Neural Network), etc. Among them, the neural network model can be a recurrent neural network (Recurrent Neural Network, RNN), a convolutional neural network (Convolution Neural Network, CNN), a long short-term memory (LSTM), an attention transformer (Attention Transformer), etc., which are not specifically limited in this application. The second training phoneme sequence is the training data of the initial language model. When the second training phoneme sequence includes second training punctuation phonemes, the first training text information includes training punctuation marks. Exemplarily, taking the second training phoneme sequence as "zh,o,ng,g,u,o,sil,h,a,o" as an example, the first training text information corresponding to this second training phoneme sequence is "China, good". Through the initial language model, the second training phoneme sequence can be converted into the first predicted text information.

[0132] Step S720: Obtain the second error information between the first predicted text information and the first training text information.

[0133] The processor compares the first predicted text information and the first training text information to obtain the second error information between the two. As an implementation, the first predicted text information can be converted into a third feature matrix, and the first training text information can be converted into a fourth feature matrix. The second error information is obtained by calculating the difference between the third feature matrix and the fourth feature matrix. Optionally, the second error information can also be determined by methods such as maximum likelihood estimation (maximum likelihoodestimation), cross-entropy loss function (cross entropy loss function), and maximum mutual information estimation (maximum mutual information estimation).

[0134] Step S730: If the second error information does not meet the second preset condition, optimize the parameters of the initial language model based on the second error information and the second optimization function, and restart the step of obtaining the first predicted text information corresponding to the second training phoneme sequence through the initial language model.

[0135] The second preset condition is used to characterize whether parameter optimization training is complete. If the second error information does not meet the second preset condition, it indicates that the error between the first predicted text information obtained through the initial language model and the first training text information is large, and the parameters in the initial language model need to be optimized and adjusted. In some embodiments, the second preset condition means that the second error information between the first predicted text information and the first training text information is less than the second preset value. The above-mentioned second preset value is actually determined according to the accuracy requirements of the language model. If the accuracy requirements of the language model are high, the second preset value is small; if the accuracy requirements of the language model are low, the second preset value is large.

[0136] The second optimization function represents a function that optimizes and adjusts the model parameters of the initial language model. The second optimization function includes, but is not limited to, batch gradient descent, stochastic gradient descent, and adaptive optimization functions. Taking the second error information as input, the second optimization function optimizes and adjusts the parameters in the initial language model so that the first predicted text information output by the optimized initial language model is closer to the first training text information.

[0137] Furthermore, the optimized and adjusted initial language model replaces the initial language model in step S710, and execution restarts from step S710.

[0138] Step S740: If the second error information meets the second preset condition, then obtain the initial language model that has completed training.

[0139] If the second error information meets the second preset condition, it means that the second error information of the first predicted text information and the first training text information obtained through the initial language model is small. At this time, the initial language model completes the model optimization training.

[0140] In some embodiments, after the language model completes optimized training, a model fine-tuning optimization step is also included. In this fine-tuning optimization step, second training text information is used for fine-tuning. The second training text information represents training text information in a new scenario, where the new scenario is different from the scenario corresponding to the first training text information. For example, the first training text information used in the language model's optimized training comes from everyday life scenarios, while the second training text information used in the language model's fine-tuning optimization comes from academic conference scenarios. By fine-tuning the language model using the second training text information from the new domain, the language model can better adapt to audio segments in the new domain.

[0141] As one implementation method, fine-tuning and optimization of the language model can be achieved through transfer learning. Specifically, if the language model is trained based on a neural network model, the learning rate of the model is reduced during transfer learning. For example, the learning rate can be reduced to 50% of the original training rate, where the original training rate is the rate at which the language model is trained. For instance, if the original training rate is 0.001, the corresponding training rate during transfer learning is adjusted to 0.0005. If the language model is trained based on a Chinese language model (N-gram), then during transfer learning, second training text information from a new domain is added to the original language model through interpolation, and the weights corresponding to the second training text information range from 0.1 to 0.4.

[0142] In this embodiment, a method for fine-tuning and optimizing a language model through transfer learning is presented, which enables the language model to have better adaptability to audio segments in different domains. At the same time, in transfer learning, only the second training text information is needed, and the audio segment corresponding to the second training text information is not required, which reduces the difficulty of obtaining training data.

[0143] This application provides a method for training a language model, which can optimize the training of the language model, resulting in a higher accuracy in recognizing text information.

[0144] Please see Figure 8 This document illustrates a structural block diagram of an audio recognition device 800 based on an acoustic model and a language model, according to an embodiment of this application. The device 800 includes: an audio segment acquisition module 810, an acoustic model processing module 820, and a language model processing module 830. The audio segment acquisition module 810 acquires audio segments. The acoustic model processing module 820 processes the audio segments based on an acoustic model to obtain a phoneme sequence corresponding to the audio segment. The phoneme sequence includes punctuation phonemes used to represent punctuation marks. The acoustic model is obtained by training an initial acoustic model using a first training audio segment. The first training audio segment is labeled with a first training phoneme sequence, which includes first training punctuation phonemes. The language model processing module 830 processes the phoneme sequence based on a language model to obtain text information corresponding to the phoneme sequence. The text information includes punctuation marks. The language model is obtained by training a preset language model using a second training phoneme sequence. The second training phoneme sequence is labeled with first training text information, which includes second training punctuation phonemes. The first training text information includes training punctuation marks.

[0145] In some embodiments, the language model processing module 830 is further configured to process the phoneme sequence based on the language model to obtain the candidate probability of at least one candidate text, wherein the candidate probability represents the probability that the candidate text is the text information corresponding to the phoneme sequence. The candidate text corresponding to the highest candidate probability is determined as the text information corresponding to the phoneme sequence.

[0146] In some embodiments, the candidate text includes at least one candidate character and at least one candidate punctuation mark, with each candidate character corresponding to a candidate character probability and each candidate punctuation mark corresponding to a candidate punctuation mark probability. The language model processing module 830 is further configured to process the phoneme sequence based on a language model to obtain at least one candidate character probability corresponding to at least one candidate text; process the phoneme sequence based on a language model to obtain at least one candidate punctuation mark probability corresponding to at least one candidate text; and determine the product of the at least one candidate character probability and the at least one candidate punctuation mark probability as the candidate probability of at least one candidate text.

[0147] In some embodiments, the language model processing module 830 is further configured to detect whether the target word in the candidate text exists in the hot word library; if the target word in the candidate text exists in the hot word library, the text candidate probability corresponding to the target word is updated, and the updated text candidate probability corresponding to the target word is greater than the text candidate probability corresponding to the target word before the update.

[0148] In some embodiments, the language model processing module 830 is further configured to determine the updated text candidate probability corresponding to the target word as the product between the text candidate probability corresponding to the target word and the first mapping factor, wherein the first mapping factor is greater than one.

[0149] In some embodiments, the language model processing module 830 is further configured to replace a first word in the candidate text with a second word based on a preset mapping table, the preset mapping table including at least one incorrect recognition result and a correct recognition result corresponding to the incorrect recognition result; the first word is an incorrect recognition result in the preset mapping table, and the second word is a correct recognition result in the preset mapping table. The text candidate probability corresponding to the first word is updated, and the updated text candidate probability corresponding to the first word is greater than the unupdated text candidate probability corresponding to the first word.

[0150] In some embodiments, the audio segment includes at least one pause segment, which represents a segment with a loudness value less than a preset loudness value and a duration greater than or equal to a preset duration. The acoustic model processing module 820 is further configured to process the audio segment based on an acoustic model to determine at least one pause segment. Based on the at least one pause segment, position information of the at least one pause segment is determined, the position information representing the position of the pause segment within the audio segment. Based on the position information, a phoneme sequence corresponding to the audio segment is obtained, the phoneme sequence including at least one punctuation phoneme for representing punctuation marks, the punctuation phoneme including quiet sounds and / or, unidentified sounds.

[0151] In some embodiments, the acoustic model processing module 820 is further configured to obtain the loudness values ​​of audio segments at different times based on an acoustic model. Audio sub-segments with loudness values ​​less than a preset loudness value are identified as candidate pause segments. If the audio duration of a candidate pause segment is greater than or equal to a preset duration, the candidate pause segment is identified as a pause segment.

[0152] In some embodiments, the apparatus 800 further includes an acoustic model training module (not shown in the figure). The acoustic model training module (not shown in the figure) is used to obtain a predicted phoneme sequence corresponding to a first training audio segment through an initial acoustic model. First error information between the predicted phoneme sequence and the first training phoneme sequence is obtained. If the first error information does not meet a first preset condition, the parameters of the initial acoustic model are optimized based on the first error information and a first optimization function, and the step of obtaining the predicted phoneme sequence corresponding to the first training audio segment through the initial acoustic model is restarted. If the first error information meets the first preset condition, the trained initial acoustic model is obtained.

[0153] In some embodiments, the apparatus 800 further includes a language model training module (not shown in the figure). The language model training module (not shown in the figure) is used to obtain first predicted text information corresponding to a second training phoneme sequence through an initial language model. It acquires second error information between the first predicted text information and the first training text information. If the second error information does not meet a second preset condition, the parameters of the initial language model are optimized based on the second error information and a second optimization function, and the step of obtaining the first predicted text information corresponding to the second training phoneme sequence through the initial language model is restarted. If the second error information meets the second preset condition, a preset language model that has completed training is acquired.

[0154] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0155] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0156] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0157] This application provides an audio recognition device based on acoustic and language models. The device determines the location information of pause segments in an audio clip using an acoustic model, and further determines a phoneme sequence including punctuation phonemes based on the determined location information. It then obtains the candidate probabilities of at least one candidate text using a language model, and identifies the candidate text with the highest candidate probability as the text information corresponding to the phoneme sequence. This device eliminates the need for additional punctuation and search modules to process plain text information; it directly obtains the text information with punctuation marks corresponding to the audio clip through the acoustic and language models, making audio recognition more convenient and efficient.

[0158] Please see Figure 9 This application also provides an electronic device 900, which includes one or more processors 910, a memory 920, and one or more application programs. The one or more application programs are stored in the memory and configured to be executed by the one or more processors, and are configured to perform the aforementioned audio recognition method based on acoustic and language models.

[0159] The processor 910 may include one or more processing cores. The processor 910 connects to various parts of the entire battery management system using various interfaces and lines, and performs various functions and processes data of the battery management system by running or executing instructions, programs, code sets, or instruction sets stored in the memory 920, and by calling data stored in the memory 920. Optionally, the processor 910 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 910 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 910 and may be implemented separately through a communication chip.

[0160] The memory 920 may include random access memory (RAM) or read-only memory (ROM). The memory 920 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created during the use of the electronic device (such as phonebook data, audio and video data, chat log data, etc.).

[0161] In some embodiments, the electronic device 900 further includes a microphone 930. The microphone 930 is a sound-picking sensor, primarily used to collect audio segments and convert them into electrical signals for transmission to the processor 910 for audio recognition. The microphone 930 can be a microphone that establishes a wired / wireless connection with the electronic device 900, a microphone assembly on an earphone that establishes a wired / wireless connection with the electronic device 900, or a microphone assembly disposed within the electronic device 900. This application embodiment does not limit the specific type of microphone 930.

[0162] In some embodiments, the electronic device 900 further includes a display device 940. The display device 940 is used to display text information with punctuation marks after the audio segment has been identified. The display device 940 may be an LED display screen, a liquid crystal display (LCD) screen, a cathode ray tube (CRT) display screen, etc. The specific type of the display device 940 is not limited in this application embodiment.

[0163] Please see Figure 10 The present application also provides a computer-readable storage medium 1000, which stores computer program instructions 1010 that can be invoked by a processor to execute the methods described in the above embodiments.

[0164] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media includes non-transitory computer-readable storage media. Computer-readable storage medium 1000 has storage space for computer program instructions 1010 that perform any of the method steps described above. These computer program instructions 1010 can be read from or written to one or more computer program products. The computer program instructions 1010 can be compressed in an appropriate form.

[0165] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. An audio recognition method based on an acoustic model and a language model, characterized by, The method comprises: acquiring an audio segment; processing the audio segment based on an acoustic model to obtain a phoneme sequence corresponding to the audio segment, the phoneme sequence comprising punctuation phonemes for representing punctuation marks, the punctuation phonemes comprising a quiet phoneme and an unknown phoneme, the quiet phoneme being obtained by processing an audio sub-segment for which the acoustic model fails to extract speech features and whose average loudness value is less than or equal to a preset loudness value, and the unknown phoneme being obtained by processing an audio sub-segment for which the acoustic model fails to extract speech features and whose average loudness value is greater than the preset loudness value, the acoustic model being obtained by training an initial acoustic model based on a first training audio segment, the first training audio segment being labeled with a first training phoneme sequence, the first training phoneme sequence comprising first training punctuation phonemes; processing the phoneme sequence based on a language model to obtain text information corresponding to the phoneme sequence, the text information comprising punctuation marks, the language model being obtained by training an initial language model based on a second training phoneme sequence, the second training phoneme sequence being labeled with first training text information, the second training phoneme sequence comprising second training punctuation phonemes, and the first training text information comprising training punctuation marks.

2. The method of claim 1, wherein, The processing of the phoneme sequence based on the language model to obtain text information corresponding to the phoneme sequence comprises: processing the phoneme sequence based on the language model to obtain candidate probabilities of at least one candidate text, the candidate probabilities representing probabilities that the candidate texts are the text information corresponding to the phoneme sequence; determining a candidate text corresponding to a maximum candidate probability as the text information corresponding to the phoneme sequence.

3. The method of claim 2, wherein, The candidate text comprises at least one candidate character and at least one candidate punctuation mark, the candidate character corresponding to a character candidate probability, and the candidate punctuation mark corresponding to a punctuation mark candidate probability. The processing of the phoneme sequence based on the language model to obtain candidate probabilities of at least one candidate text comprises: processing the phoneme sequence based on the language model to obtain at least one character candidate probability corresponding to at least one candidate text; processing the phoneme sequence based on the language model to obtain at least one punctuation mark candidate probability corresponding to at least one candidate text; determining a product of the at least one character candidate probability and the at least one punctuation mark candidate probability as the candidate probability of the at least one candidate text.

4. The method of claim 3, wherein, After the processing of the phoneme sequence based on the language model to obtain at least one character candidate probability corresponding to at least one candidate text, the method further comprises: detecting whether a target term in the candidate text exists in a hot word library; if the target term in the candidate text exists in the hot word library, updating a character candidate probability corresponding to the target term, the updated character candidate probability corresponding to the target term being greater than the character candidate probability corresponding to the target term before the update.

5. The method of claim 4, wherein, If the target word in the candidate text exists in the hot word library, the literal candidate probability corresponding to the target word is updated, including: The product of the literal candidate probability corresponding to the target word and a first mapping factor is determined as the updated literal candidate probability corresponding to the target word, and the first mapping factor is greater than one.

6. The method of claim 3, wherein, After the language model is used to process the phoneme sequence to obtain at least one literal candidate probability corresponding to at least one candidate text, the method further includes: A first word in the candidate text is replaced by a second word based on a preset mapping table, the preset mapping table including at least one incorrect recognition result and a correct recognition result corresponding to the incorrect recognition result; the first word is the incorrect recognition result in the preset mapping table, and the second word is the correct recognition result in the preset mapping table; The literal candidate probability corresponding to the first word is updated, and the updated literal candidate probability corresponding to the first word is greater than the literal candidate probability corresponding to the first word before the update.

7. The method of claim 1, wherein, The audio segment includes at least one pause segment, the pause segment representing a segment with a loudness value less than a preset loudness value and a time length greater than or equal to a preset time length, and the acoustic model is used to process the audio segment to obtain a phoneme sequence corresponding to the audio segment, including: The acoustic model is used to process the audio segment to determine at least one pause segment; Based on at least one pause segment, position information of at least one pause segment is determined, the position information representing a position of the pause segment in the audio segment; Based on the position information, a phoneme sequence corresponding to the audio segment is obtained, the phoneme sequence including at least one punctuation phoneme representing a punctuation symbol, and the punctuation phoneme including a silence phoneme and / or an unknown phoneme.

8. The method of claim 7, wherein the acoustic model is used to process the audio segment to determine at least one pause segment, including: Based on the acoustic model, a loudness value corresponding to the audio segment at different time instants is obtained; An audio sub-segment with a loudness value less than the preset loudness value in the audio segment is determined as a candidate pause segment; If the time length of the candidate pause segment is greater than or equal to the preset time length, the candidate pause segment is determined as the pause segment.

9. The method according to any one of claims 1 to 8, characterized in that, The training process of the acoustic model is as follows: A predicted phoneme sequence corresponding to the first training audio segment is obtained through an initial acoustic model; First error information between the predicted phoneme sequence and the first training phoneme sequence is obtained; If the first error information does not meet a first preset condition, the initial acoustic model is parameter-optimized based on the first error information and a first optimization function, and the step of obtaining the predicted phoneme sequence corresponding to the first training audio segment through the initial acoustic model is re-executed; If the first error information meets the first preset condition, the initial acoustic model that has completed training is obtained.

10. The method according to any one of claims 1 to 8, characterized in that, The training process of the language model is as follows: obtaining first predicted text information corresponding to the second training phoneme sequence through the initial language model; obtaining second error information between the first predicted text information and the first training text information; if the second error information does not conform to a second preset condition, performing parameter optimization on the initial language model based on the second error information and a second optimization function, and starting to execute the step of obtaining the first predicted text information corresponding to the second training phoneme sequence through the initial language model again; if the second error information conforms to the second preset condition, obtaining a preset language model trained completely.

Images