Audio recognition method, apparatus, device, computer-readable medium, and program product
By utilizing pre-trained acoustic and language models during the audio recognition process to remove text length anomalies and determine the optimal text length, the problem of acoustic and language models being affected by accents and noise is solved, achieving more accurate and efficient audio recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JD DIGITS HAIYI INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2024-12-17
- Publication Date
- 2026-06-19
Smart Images
Figure CN122245299A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to audio recognition methods, apparatus, devices, computer-readable media, and program products. Background Technology
[0002] Currently, with the rapid development of artificial intelligence and deep learning technologies, Automatic Speech Recognition (ASR) systems have been widely applied in e-commerce, logistics, finance, and many other fields. As their application becomes more widespread, ASR systems not only play a crucial role in customer service and interactive experience, but also bring more development opportunities for improving work efficiency, reducing costs, and creating new business models. For ASR-based audio recognition, the typical approach is as follows: first, an acoustic model is used to identify and understand the acoustic features in the speech, and a language model is used to understand the context and grammatical rules of the language, generating the corresponding recognized text.
[0003] However, the inventors discovered that when using the above method to generate audio-corresponding text, the following technical problems often arise:
[0004] Acoustic and language models are easily affected by factors such as accents and background noise, which can lead to inaccurate recognition of the text.
[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.
[0007] Some embodiments of this disclosure provide audio recognition methods, apparatuses, devices, computer-readable media, and program products to address the technical problems mentioned in the background section above.
[0008] In a first aspect, some embodiments of this disclosure provide an audio recognition method, comprising: generating a candidate text set for target audio feature information using a pre-trained acoustic model, wherein the target audio feature information corresponds to the audio to be recognized; determining text length anomaly information corresponding to each candidate text in the candidate text set, thereby obtaining a text length anomaly information set; in response to determining that there is text length anomaly information representing text length anomaly in the text length anomaly information set, removing the corresponding candidate text representing text length anomaly from the candidate text set, thereby obtaining a removed text set; determining an optimal text length for the candidate text set; and generating the recognized text corresponding to the audio using a pre-trained language model based on the optimal text length and the removed text set.
[0009] Optionally, determining the text length anomaly information corresponding to each candidate text in the candidate text set includes: determining the text length corresponding to each candidate text in the candidate text set to obtain a text length set; determining the length mean and length standard deviation corresponding to the text length set; determining the text length corresponding to the candidate text as the target text length; determining the difference between the target text length and the length mean; and generating the text length anomaly information corresponding to the candidate text based on the difference and the length standard deviation.
[0010] Optionally, determining the optimal text length for the candidate text set includes: obtaining an acoustic model evaluation information set for the candidate text set, wherein there is a one-to-one correspondence between the candidate texts and the acoustic model evaluation information, and the acoustic model evaluation information is generated based on the acoustic model; and generating the optimal text length according to the acoustic model evaluation information set and a pre-set text length regularization term.
[0011] Optionally, the above-mentioned method of generating the recognition text corresponding to the audio using a pre-trained language model based on the optimal text length and the removed text set includes: selecting texts with the same text length as the optimal text length from the removed text set to obtain a selected text set; for each text in the selected text set, performing the following first generation step: determining the acoustic model evaluation information and language model evaluation information corresponding to the text, wherein the language model evaluation information is generated based on the language model; generating comprehensive model evaluation information for the text based on the acoustic model evaluation information and the language model evaluation information; and selecting texts from the selected text set whose comprehensive model evaluation information satisfies a first preset evaluation condition as recognition text.
[0012] Optionally, the method further includes: in response to determining that there is no text length anomaly information representing text length anomaly in the above text length anomaly information set, for each candidate text in the above candidate text set, performing the following second generation step: determining the acoustic model evaluation information and language model evaluation information corresponding to the above candidate text; generating model comprehensive evaluation information for the above text based on the acoustic model evaluation information and the language model evaluation information; and selecting texts from the above candidate text set whose corresponding model comprehensive evaluation information meets the second preset evaluation condition as the identified text.
[0013] Optionally, the acoustic model evaluation information is an acoustic model evaluation score, and the language model evaluation information is a language model evaluation score; and generating comprehensive model evaluation information for the text based on the acoustic model evaluation information and the language model evaluation information includes: performing a weighted summation on the acoustic model evaluation score and the language model evaluation score to generate a weighted summation score as the comprehensive model evaluation information; and selecting texts from the filtered text set whose comprehensive model evaluation information meets the first preset evaluation condition as identification texts includes: selecting the text with the highest corresponding weighted summation score from the filtered text set as identification texts.
[0014] Secondly, some embodiments of this disclosure provide an audio recognition device, comprising: a first generation unit configured to generate a candidate text set for target audio feature information using a pre-trained acoustic model, wherein the target audio feature information corresponds to the audio to be recognized; a first determination unit configured to determine text length anomaly information corresponding to each candidate text in the candidate text set, thereby obtaining a text length anomaly information set; a removal unit configured to remove candidate texts corresponding to text length anomalies from the candidate text set in response to determining that there are text length anomaly information representing text length anomalies in the text length anomaly information set, thereby obtaining a removed text set; a second determination unit configured to determine an optimal text length for the candidate text set; and a second generation unit configured to generate recognition text corresponding to the audio using a pre-trained language model based on the optimal text length and the removed text set.
[0015] Optionally, the first determining unit can be configured to: determine the text length corresponding to each candidate text in the above candidate text set to obtain a text length set; determine the length mean and length standard deviation corresponding to the above text length set; determine the text length corresponding to the above candidate text as the target text length; determine the difference between the above target text length and the above length mean; and generate text length anomaly information corresponding to the above candidate text based on the above difference and the above length standard deviation.
[0016] Optionally, the second determining unit can be configured to: obtain an acoustic model evaluation information set for the above candidate text set, wherein there is a one-to-one correspondence between the candidate text and the acoustic model evaluation information, and the acoustic model evaluation information is generated based on the above acoustic model; and generate the above optimal text length according to the above acoustic model evaluation information set and a pre-set text length regularization term.
[0017] Optionally, the second generation unit can be configured to: select texts from the removed text set whose corresponding text length is the same as the optimal text length, to obtain a filtered text set; for each text in the filtered text set, perform the following first generation step: determine the acoustic model evaluation information and language model evaluation information corresponding to the text, wherein the language model evaluation information is generated based on the language model; generate a comprehensive model evaluation information for the text based on the acoustic model evaluation information and the language model evaluation information; select texts from the filtered text set whose comprehensive model evaluation information satisfies the first preset evaluation condition, as the recognition text.
[0018] Optionally, the apparatus further includes: in response to determining that there is no text length anomaly information representing a text length anomaly in the aforementioned text length anomaly information set, for each candidate text in the aforementioned candidate text set, performing the following second generation step: determining the acoustic model evaluation information and language model evaluation information corresponding to the aforementioned candidate text; generating model comprehensive evaluation information for the aforementioned text based on the aforementioned acoustic model evaluation information and the aforementioned language model evaluation information; and selecting texts from the aforementioned candidate text set whose corresponding model comprehensive evaluation information satisfies a second preset evaluation condition as identified texts.
[0019] Optionally, the acoustic model evaluation information is an acoustic model evaluation score, and the language model evaluation information is a language model evaluation score; and the second generation unit can be configured to: perform a weighted summation process on the acoustic model evaluation score and the language model evaluation score to generate a weighted summation score, which serves as the comprehensive evaluation information of the model. The text with the highest corresponding weighted summation score is selected from the filtered text set and used as the recognized text.
[0020] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.
[0021] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.
[0022] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.
[0023] The above embodiments of this disclosure have the following beneficial effects: By removing candidate texts with abnormal text lengths, the audio recognition method of some embodiments of this disclosure can accurately and efficiently generate the recognition text corresponding to the audio. Specifically, the reason why the related recognition text is not accurate enough is that the acoustic model and language model are easily affected by factors such as accents and background noise, resulting in inaccurate recognition text. Based on this, the audio recognition method of some embodiments of this disclosure firstly uses a pre-trained acoustic model to initially and accurately generate a candidate text set targeting the target audio feature information. The target audio feature information corresponds to the audio to be recognized. Here, the obtained candidate text is the candidate text for which the accuracy of the text content is to be determined (i.e., whether it is an accurate recognition text corresponding to the audio). Then, the text length abnormality information corresponding to each candidate text in the above candidate text set is determined, resulting in a text length abnormality information set. Here, by determining the text length corresponding to each candidate text, the poor-quality candidate text generated due to the influence of factors such as accents and background noise can be effectively removed. Next, in response to the determination that there are text length anomalies in the aforementioned text length anomaly information set, the corresponding candidate texts representing text length anomalies are removed from the aforementioned candidate text set, resulting in a removed text set. Here, by removing candidate texts with text length anomalies, the accuracy of subsequent candidate text determination can be reduced, thus reducing the determination time for the identified text. Furthermore, the optimal text length for the aforementioned candidate text set is determined. Here, by limiting the optimal text length corresponding to the candidate text set, subsequent determination can focus on the text content of candidate texts with lengths close to the optimal text length, thereby improving the accuracy of the identified text. Finally, based on the aforementioned optimal text length and the aforementioned removed text set, the identified text corresponding to the aforementioned audio can be accurately generated using a pre-trained language model. In summary, based on removing candidate texts with text length anomalies and determining the optimal text length, the identified text corresponding to the audio can be generated accurately and efficiently. Attached Figure Description
[0024] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.
[0025] Figure 1This is a schematic diagram illustrating an application scenario of an audio recognition method according to some embodiments of the present disclosure;
[0026] Figure 2 This is a flowchart of some embodiments of the audio recognition method according to the present disclosure;
[0027] Figure 3 This is a flowchart of some other embodiments of the audio recognition method according to the present disclosure;
[0028] Figure 4 This is a schematic diagram of the structure of some embodiments of the audio recognition device according to the present disclosure;
[0029] Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation
[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0031] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.
[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0035] Before performing any of the operations involving the collection, storage, and use of user personal information (such as audio to be recognized by text), the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing personal information subjects, and obtaining prior authorization and consent from personal information subjects.
[0036] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.
[0037] Figure 1 This is a schematic diagram of an application scenario of an audio recognition method according to some embodiments of the present disclosure.
[0038] exist Figure 1 In this application scenario, firstly, the electronic device 101 can use a pre-trained acoustic model 102 to generate a candidate text set 105 targeting the audio feature information 104. The target audio feature information 104 corresponds to the audio 103 to be recognized. In this application scenario, the candidate text set 105 may include: text 1, text 2, text 3, and text 4. Then, the electronic device 101 can determine the text length anomaly information corresponding to each candidate text in the candidate text set 105, obtaining a text length anomaly information set. Next, in response to determining that there is text length anomaly information representing a text length anomaly in the text length anomaly information set, the electronic device 101 can remove the corresponding candidate text representing the text length anomaly from the candidate text set 105, obtaining a removed text set 106. In this application scenario, text 2 represents the text representing the text length anomaly. The removed text set 106 may include: text 1, text 3, and text 4. Furthermore, the electronic device 101 can determine the optimal text length 107 for the candidate text set 105. In this application scenario, the optimal text length 107 can be "8". Finally, the electronic device 101 can generate the recognition text 110 corresponding to the audio 103 based on the optimal text length 107 and the removed text set 106, using the pre-trained language model 108. In this application scenario, the recognition text 110 can be text 3.
[0039] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.
[0040] It should be understood that Figure 1The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.
[0041] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of an audio recognition method according to the present disclosure. The audio recognition method includes the following steps:
[0042] Step 201: Using a pre-trained acoustic model, generate a candidate text set targeting the target audio feature information.
[0043] In some embodiments, the entity executing the above-described audio recognition method (e.g.) Figure 1 The electronic device 101 shown can utilize a pre-trained acoustic model to generate a candidate text set targeting audio feature information. The acoustic model can be a pre-trained neural network model used to identify and understand acoustic features in audio (e.g., spectrum, duration, and frequency of sound) to output the candidate text set. In practice, the acoustic model can be an acoustic model (AM) in an automatic speech recognition model. The target audio feature information can be information representing the semantic content of the audio features corresponding to the audio to be recognized. The aforementioned target audio feature information corresponds to the audio to be recognized. The candidate text can be the text to be determined as the audio content corresponding to the audio (i.e., the recognized text). In practice, the target audio feature information can be feature information in matrix form. The corresponding target audio feature information can be a T*D audio feature matrix. Here, T represents the audio feature frame length, and D is the audio feature dimension. In a specific practical scenario, the acoustic model is a model obtained by fine-tuning a 160,000-hour Chinese dataset based on a large language recognition model, with a total parameter count of 1.5B. The audio feature matrix is a D=128-dimensional Mel-spectral feature. Each frame has a duration of 25ms and a step size of 10ms.
[0044] As an example, the aforementioned execution entity can input the target audio feature information into the acoustic model to generate a candidate text set.
[0045] Step 202: Determine the text length anomaly information corresponding to each candidate text in the above candidate text set to obtain a text length anomaly information set.
[0046] In some embodiments, the executing entity can determine the text length anomaly information corresponding to each candidate text in the candidate text set, thereby obtaining a text length anomaly information set. The text length anomaly information can characterize whether the text length corresponding to the candidate text is abnormal. For example, the text length anomaly information can be "0" or "1". When the text length anomaly information is "0", it indicates that the text length corresponding to the candidate text is abnormal. When the text length anomaly information is "1", it indicates that the text length corresponding to the candidate text is normal. The text length can be the character length of each character included in the candidate text.
[0047] As an example, firstly, the aforementioned execution entity can determine the text length corresponding to each candidate text in the candidate text set, obtaining a text length set. Then, the text lengths in the text length set are sorted in ascending order to generate a text length sequence. Next, a subsequence of text lengths corresponding to the central sequence position and having a predetermined length is selected from the text length sequence. Then, each text length in the subsequence is determined as a normal length, and the remaining text lengths after removing the subsequence are determined as abnormal lengths. Finally, based on the determined normal and abnormal lengths, text length abnormality information corresponding to each candidate text in the aforementioned candidate text set is determined, resulting in a text length abnormality information set.
[0048] In some optional implementations of certain embodiments, the execution entity may determine the text length anomaly information corresponding to each candidate text in the candidate text set, including the following steps:
[0049] The first step is to determine the text length corresponding to each candidate text in the above candidate text set, thus obtaining the text length set.
[0050] As an example, the aforementioned execution entity can determine the text length corresponding to each candidate text in the aforementioned candidate text set by determining the number of characters, thereby obtaining a text length set.
[0051] The second step is to determine the mean and standard deviation of the text lengths corresponding to the above text length set. The mean length can be the average of the lengths of all texts in the text length set. The standard deviation of the lengths can be the standard deviation of the lengths of all texts in the text length set.
[0052] The third step is to determine the text length corresponding to the above candidate texts, which will be used as the target text length.
[0053] The fourth step is to determine the difference between the target text length and the average length. This difference can be a value greater than 0.
[0054] As an example, the aforementioned execution entity can first subtract the average length from the target text length to generate a subtracted value. Then, the absolute value of the subtracted value is processed to generate a difference.
[0055] Fifth, based on the above differences and the above length standard deviation, generate the text length anomaly information corresponding to the above candidate texts.
[0056] As an example, firstly, the aforementioned execution entity can multiply the length standard deviation by the target value to generate a multiplied value. The target value can be a positive integer. For example, a target value of 1.65 corresponds to the 90th percentile of a Gaussian distribution. Then, in response to determining that the difference is greater than the multiplied value, text length anomaly information is generated, indicating that the text length of the candidate text is abnormal. In response to determining that the difference is less than or equal to the multiplied value, text length anomaly information is generated, indicating that the text length of the candidate text is not abnormal.
[0057] As another example, in response to determining that the difference is greater than the length standard deviation, text length anomaly information is generated indicating that the text length of the candidate text is abnormal. In response to determining that the difference is less than or equal to the length standard deviation, text length anomaly information is generated indicating that the text length of the candidate text is not abnormal.
[0058] Step 203: In response to determining that there is text length anomaly information in the above text length anomaly information set that represents text length anomaly, remove the corresponding candidate text representing text length anomaly from the above candidate text set to obtain the removed text set.
[0059] In some embodiments, in response to determining that there is text length anomaly information in the above-mentioned text length anomaly information set that represents text length anomaly, the above-mentioned execution entity can remove the corresponding candidate text representing text length anomaly from the above-mentioned candidate text set to obtain the removed text set.
[0060] As an example, the aforementioned execution entity can set the acoustic model evaluation information and language model evaluation information corresponding to candidate texts that represent abnormal text length to empty in order to remove candidate texts that represent abnormal text length.
[0061] Step 204: Determine the optimal text length for the above candidate text set.
[0062] In some embodiments, the execution entity may determine the optimal text length for the candidate text set. The optimal text length may be the most suitable text length representing the audio corresponding to the text to be recognized. For example, if the optimal text length is 7, it represents the corresponding recognized text length as 7. That is, the optimal text length is the same as the text length corresponding to the subsequent recognized text.
[0063] As an example, firstly, the aforementioned execution entity can determine the text length subsequence corresponding to the candidate text set. Then, it averages the text lengths in the aforementioned text length subsequence to generate an average text length, which is then used as the optimal text length.
[0064] In some optional implementations of certain embodiments, the execution entity determines the optimal text length for the candidate text set by including the following steps:
[0065] The first step is to obtain an acoustic model evaluation information set for the aforementioned candidate text set. There is a one-to-one correspondence between the candidate texts and the acoustic model evaluation information. The acoustic model evaluation information is generated based on the aforementioned acoustic model. The acoustic model evaluation information can be an evaluation of the accuracy of the acoustic model on the corresponding text content of the candidate text. In practice, the acoustic model evaluation information can be evaluation labels in text form. For example, the acoustic model evaluation information can be, but is not limited to, one of the following: poor acoustic model evaluation information, moderate acoustic model evaluation information, good acoustic model evaluation information, and very good acoustic model evaluation information. Poor acoustic model evaluation information can characterize that the accuracy of the corresponding candidate text's text content is poor. Moderate acoustic model evaluation information can characterize that the accuracy of the corresponding candidate text's text content is moderate. Good acoustic model evaluation information can characterize that the accuracy of the corresponding candidate text's text content is good. Very good acoustic model evaluation information can characterize that the accuracy of the corresponding candidate text's text content is very good. Each label-based acoustic model evaluation information can also have a corresponding score. In addition, the acoustic model evaluation information can also be in the form of scores. The higher the corresponding score, the better the accuracy of the text content corresponding to the candidate text.
[0066] The second step is to generate the optimal text length based on the acoustic model evaluation information set and the pre-set text length regularization term.
[0067] As an example, the above-mentioned execution entity can determine the optimal text length using the following formula:
[0068] L o =argmax n {log(p AM (y n |x))-λlog({L n )},
[0069] Among them, L o This is the optimal text length. AM (y n |x) represents the score corresponding to the acoustic model evaluation information. y n The nth candidate text. X represents the target audio feature information. λ is the weight parameter corresponding to the text length regularization term. L nλ represents the length of the nth candidate text. λ can be a pre-set weight value. For example, λ could be the value 2.
[0070] As an example, firstly, the aforementioned execution entity can filter at least one candidate text from the candidate text set whose corresponding acoustic model evaluation information is either "good acoustic model evaluation information" or "very good acoustic model evaluation information." Then, it determines the first regularization term value corresponding to the "good acoustic model evaluation information" and the second regularization term value corresponding to the "very good acoustic model evaluation information." Next, it performs a weighted sum of at least one text length corresponding to at least one candidate text and the corresponding first and second regularization term values to generate the optimal text length.
[0071] Step 205: Based on the optimal text length and the removed text set, generate the recognition text corresponding to the audio using a pre-trained language model.
[0072] In some embodiments, the execution entity can generate the recognized text corresponding to the audio based on the optimal text length and the removed text set, using a pre-trained language model. The language model can be a model that understands context and grammatical rules of language to predict the most likely words or phrases in a given context. In practice, the language model can be a Transformer language model. The recognized text can be the audio content in text form corresponding to the audio. In practice, the language model can be a language model commonly used in ASR recognition models.
[0073] As an example, firstly, texts whose length difference from the optimal text length is less than a predetermined value are selected from the removed text set to obtain the target candidate text set. Then, using a language model, text evaluation information is determined for each target candidate text in the target candidate text set, resulting in a text evaluation information set. This text evaluation information characterizes the degree of matching between the text content and the audio content of the target candidate text. The text evaluation information can include: evaluation tags + probabilities. Evaluation tags can include: tags indicating low matching degree, tags indicating medium matching degree, and tags indicating high matching degree. Probabilities characterize the accuracy of the evaluation tags. Finally, the target candidate text with the best corresponding evaluation tags and probabilities is selected from the target candidate text set as the recognition text.
[0074] In some optional implementations of certain embodiments, the execution entity can generate the recognized text corresponding to the audio based on the optimal text length and the removed text set, using a pre-trained language model, including the following steps:
[0075] The first step is to filter out the texts with the same length as the optimal text length from the removed text set, thus obtaining the filtered text set.
[0076] The second step involves performing the following first generation step for each text in the filtered text set:
[0077] Sub-step 1 involves determining the acoustic model evaluation information and language model evaluation information corresponding to the aforementioned text. The language model evaluation information is generated based on the aforementioned language model. Language model evaluation information can be the information obtained after inputting the filtered text pairs into the language model. Language model evaluation information can be the language model's evaluation of the accuracy of the text content corresponding to the candidate text. In practice, language model evaluation information can be evaluation labels in text form. For example, language model evaluation information can be, but is not limited to, one of the following: poor language model evaluation information, moderate language model evaluation information, good language model evaluation information, and very good language model evaluation information. Poor language model evaluation information can characterize that the accuracy of the corresponding candidate text's text content is poor. Moderate language model evaluation information can characterize that the accuracy of the corresponding candidate text's text content is moderate. Good language model evaluation information can characterize that the accuracy of the corresponding candidate text's text content is good. Very good language model evaluation information can characterize that the accuracy of the corresponding candidate text's text content is very good. Each label-based language model evaluation information can have a corresponding score. In addition, language model evaluation information can also be information in the form of scores. The higher the corresponding score, the better the accuracy of the text content corresponding to the candidate text.
[0078] Sub-step 2: Based on the above language model evaluation information, generate comprehensive model evaluation information for the above text.
[0079] As an example, the aforementioned executing entity can summarize the language model evaluation information and the evaluation information in the form of various text labels corresponding to the language model evaluation information to determine the actual model evaluation information corresponding to the text, which can then be used as the comprehensive model evaluation information.
[0080] The third step involves selecting texts from the filtered text set whose comprehensive evaluation information for the corresponding model meets the first preset evaluation condition, and using these as the identified texts. The first preset evaluation condition can be that the evaluation label corresponding to the comprehensive evaluation information of the corresponding model is higher than the target label. For example, the language model evaluation information of the text is higher than the moderate language model evaluation information, and the corresponding acoustic model evaluation information is higher than the moderate acoustic model evaluation information.
[0081] Optionally, the acoustic model evaluation information mentioned above refers to acoustic model evaluation scores, and the language model evaluation information mentioned above refers to language model evaluation scores. The acoustic model evaluation score can represent the accuracy of the corresponding text content. A higher score indicates that the corresponding content is more accurate under the acoustic model evaluation. The language model evaluation score can also represent the accuracy of the corresponding text content. A higher score indicates that the corresponding content is more accurate under the language model evaluation. The acoustic model evaluation score can be a value between 0 and 1, or a value between 0 and 100. When the value is between 0 and 1, the corresponding acoustic model evaluation information can be an acoustic model score output for each candidate text based on the acoustic model likelihood probability.
[0082] Optionally, the aforementioned execution entity may generate comprehensive model evaluation information for the aforementioned text based on the aforementioned acoustic model evaluation information and the aforementioned language model evaluation information, including the following steps:
[0083] The evaluation scores of the acoustic model and the language model are weighted and summed to generate a weighted sum score, which serves as the comprehensive evaluation information for the models.
[0084] The aforementioned executing entity can select texts from the filtered text set that meet the first preset evaluation criteria for the corresponding model comprehensive evaluation information, and use them as identification texts, including the following steps:
[0085] The aforementioned executing entity can select the text with the highest weighted sum score from the filtered text set as the identification text.
[0086] The above embodiments of this disclosure have the following beneficial effects: The audio recognition method of some embodiments of this disclosure can accurately and efficiently generate the recognized text corresponding to the audio by removing candidate texts with abnormal text lengths. Specifically, the reason for the inaccuracy of the related recognized text is that during the process of the acoustic model generating the first text evaluation information set and the language model generating the second text evaluation information set, it is easily affected by factors such as accents and background noise, resulting in the inaccuracy of the generated first and second text evaluation information sets, thus making the subsequently recognized text inaccurate. Based on this, the audio recognition method of some embodiments of this disclosure firstly uses a pre-trained acoustic model to initially and accurately generate a candidate text set targeting the target audio feature information. The target audio feature information corresponds to the audio to be recognized. Here, the obtained candidate text is the candidate text for which the accuracy of the text content is to be determined (i.e., whether it is an accurate recognized text corresponding to the audio). Then, the text length abnormality information corresponding to each candidate text in the above candidate text set is determined, resulting in a text length abnormality information set. Here, by determining the text length corresponding to each candidate text, the poor-quality candidate text generated due to the influence of factors such as accents and background noise can be effectively removed. Next, in response to the determination that there are text length anomalies in the aforementioned text length anomaly information set, the corresponding candidate texts representing text length anomalies are removed from the aforementioned candidate text set, resulting in a removed text set. Here, by removing candidate texts with text length anomalies, the accuracy of subsequent candidate text determination can be reduced, thus reducing the determination time for the identified text. Furthermore, the optimal text length for the aforementioned candidate text set is determined. Here, by limiting the optimal text length corresponding to the candidate text set, subsequent determination can focus on the text content of candidate texts with lengths close to the optimal text length, thereby improving the accuracy of the identified text. Finally, based on the aforementioned optimal text length and the aforementioned removed text set, the identified text corresponding to the aforementioned audio can be accurately generated using a pre-trained language model. In summary, based on removing candidate texts with text length anomalies and determining the optimal text length, the identified text corresponding to the audio can be generated accurately and efficiently.
[0087] Further reference Figure 3 The diagram illustrates a flow 300 of another embodiment of the audio recognition method according to the present disclosure. This audio recognition method includes the following steps:
[0088] Step 301: Using a pre-trained acoustic model, generate a candidate text set targeting the target audio feature information.
[0089] Step 302: Determine the text length anomaly information corresponding to each candidate text in the above candidate text set to obtain a text length anomaly information set.
[0090] Step 303: In response to determining that there is text length anomaly information in the above text length anomaly information set that represents text length anomaly, remove the corresponding candidate text representing text length anomaly from the above candidate text set to obtain the removed text set.
[0091] Step 304: Determine the optimal text length for the above candidate text set.
[0092] Step 305: Based on the optimal text length and the removed text set, generate the recognition text corresponding to the audio using a pre-trained language model.
[0093] Step 306: In response to determining that there is no text length anomaly information representing a text length anomaly in the above-mentioned text length anomaly information set, for each candidate text in the above-mentioned candidate text set, the following second generation step is performed:
[0094] Step 3061: Determine the acoustic model evaluation information and language model evaluation information corresponding to the above candidate texts.
[0095] In some embodiments, the executing entity (e.g. Figure 1 The electronic device 101 shown can determine the acoustic model evaluation information and language model evaluation information corresponding to the above candidate texts.
[0096] Step 3062: Based on the acoustic model evaluation information and the language model evaluation information, generate comprehensive model evaluation information for the text.
[0097] In some embodiments, the execution entity may generate comprehensive model evaluation information for the text based on the acoustic model evaluation information and the language model evaluation information.
[0098] As an example, the aforementioned executing entity can perform a weighted summation of the scores corresponding to the acoustic model evaluation information and the language model evaluation information to generate comprehensive model evaluation information.
[0099] Step 307: Select texts from the above candidate text set whose corresponding model comprehensive evaluation information meets the second preset evaluation conditions, and use them as the identification texts.
[0100] In some embodiments, the execution entity may select texts from the candidate text set whose corresponding model comprehensive evaluation information meets the second preset evaluation conditions as identification texts.
[0101] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3The corresponding audio recognition method flow 300 in some embodiments. If it is determined that the text length is not abnormal, the text to be recognized can be accurately filtered from the candidate text set.
[0102] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an audio recognition device, which are similar to... Figure 2 Corresponding to the method embodiments shown, the audio recognition device can be specifically applied to various electronic devices.
[0103] like Figure 4 As shown, an audio recognition device 400 includes: a first generation unit 401, a first determination unit 402, a removal unit 403, a second determination unit 404, and a second generation unit 405. The first generation unit 401 is configured to generate a candidate text set targeting target audio feature information using a pre-trained acoustic model, wherein the target audio feature information corresponds to the audio to be recognized. The first determination unit 402 is configured to determine text length anomaly information corresponding to each candidate text in the candidate text set, obtaining a text length anomaly information set. The removal unit 403 is configured to remove the corresponding candidate text representing text length anomaly from the candidate text set in response to determining that there is text length anomaly information representing text length anomaly in the text length anomaly information set, obtaining a removed text set. The second determination unit 404 is configured to determine the optimal text length for the candidate text set. The second generation unit 405 is configured to generate the recognition text corresponding to the audio using a pre-trained language model based on the optimal text length and the removed text set.
[0104] In some optional implementations of some embodiments, the first determining unit 402 may be further configured to: determine the text length corresponding to each candidate text in the candidate text set to obtain a text length set; determine the length mean and length standard deviation corresponding to the text length set; determine the text length corresponding to the candidate text as the target text length; determine the difference between the target text length and the length mean; and generate text length anomaly information corresponding to the candidate text based on the difference and the length standard deviation.
[0105] In some optional implementations of some embodiments, the second determining unit 404 may be further configured to: obtain an acoustic model evaluation information set for the candidate text set, wherein there is a one-to-one correspondence between the candidate text and the acoustic model evaluation information, and the acoustic model evaluation information is generated based on the acoustic model; and generate the optimal text length according to the acoustic model evaluation information set and a pre-set text length regularization term.
[0106] In some optional implementations of certain embodiments, the second generation unit 405 may be further configured to: filter out texts with the same length as the optimal text length from the removed text set to obtain a filtered text set; for each text in the filtered text set, perform the following first generation step: determine the acoustic model evaluation information and language model evaluation information corresponding to the text, wherein the language model evaluation information is generated based on the language model; generate a comprehensive model evaluation information for the text based on the acoustic model evaluation information and the language model evaluation information; and filter out texts from the filtered text set whose comprehensive model evaluation information satisfies the first preset evaluation condition as identification text.
[0107] In some optional implementations of certain embodiments, the apparatus 400 further includes a first execution unit and a first filtering unit (not shown in the figure). The first execution unit can be configured to: in response to determining that no text length anomaly information representing a text length anomaly exists in the text length anomaly information set, for each candidate text in the candidate text set, perform the following second generation step: determine the acoustic model evaluation information and language model evaluation information corresponding to the candidate text; and generate a comprehensive model evaluation information for the text based on the acoustic model evaluation information and the language model evaluation information. The first filtering unit can be configured to: filter texts from the candidate text set whose corresponding comprehensive model evaluation information satisfies a second preset evaluation condition, and use them as identification texts.
[0108] In some optional implementations of certain embodiments, the acoustic model evaluation information is an acoustic model evaluation score, and the language model evaluation information is a language model evaluation score; and the second generation unit 405 can be further configured to: perform a weighted summation process on the acoustic model evaluation score and the language model evaluation score to generate a weighted summation score, which serves as the comprehensive model evaluation information. The second generation unit 405 can be further configured to: select the text with the highest corresponding weighted summation score from the filtered text set, which serves as the recognized text.
[0109] It is understandable that the units described in the audio recognition device 400 are related to the reference. Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the audio recognition device 400 and the units contained therein, and will not be repeated here.
[0110] The following is for reference. Figure 5 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1 A schematic diagram of the structure of electronic device 101)500. Figure 5The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.
[0111] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage device 508 into a random access memory 503. The random access memory 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, the read-only memory 502, and the random access memory 503 are interconnected via a bus 504. An input / output interface 505 is also connected to the bus 504.
[0112] Typically, the following devices can be connected to the input / output interface 505: input devices 506 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.
[0113] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a read-only memory 502. When the computer program is executed by the processing device 501, it performs the functions defined above in the methods of some embodiments of this disclosure.
[0114] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0115] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0116] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: generate a candidate text set targeting target audio feature information using a pre-trained acoustic model, wherein the target audio feature information corresponds to the audio to be recognized; determine text length anomaly information corresponding to each candidate text in the candidate text set, obtaining a text length anomaly information set; in response to determining that there is text length anomaly information representing text length anomaly in the text length anomaly information set, remove the corresponding candidate text representing text length anomaly from the candidate text set, obtaining a removed text set; determine the optimal text length for the candidate text set; and, based on the optimal text length and the removed text set, generate the recognized text corresponding to the audio using a pre-trained language model.
[0117] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0119] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first generation unit, a first determination unit, a removal unit, a second determination unit, and a second generation unit. The names of these units do not necessarily limit the specific unit; for example, the second determination unit may also be described as "a unit for determining the optimal text length for the aforementioned candidate text set."
[0120] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0121] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described audio recognition methods.
[0122] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. An audio recognition method, comprising: Using a pre-trained acoustic model, a candidate text set is generated based on the target audio feature information, wherein the target audio feature information corresponds to the audio of the text to be identified; Determine the text length anomaly information corresponding to each candidate text in the candidate text set to obtain a text length anomaly information set; In response to determining that there is text length anomaly information in the text length anomaly information set that represents a text length anomaly, the corresponding candidate text representing the text length anomaly is removed from the candidate text set to obtain the removed text set; Determine the optimal text length for the candidate text set; Based on the optimal text length and the removed text set, a pre-trained language model is used to generate the recognition text corresponding to the audio.
2. The method according to claim 1, wherein, The step of determining the text length anomaly information corresponding to each candidate text in the candidate text set includes: Determine the text length corresponding to each candidate text in the candidate text set to obtain a text length set; Determine the mean length and standard deviation of the length corresponding to the text length set; Determine the text length corresponding to the candidate text, and use it as the target text length; Determine the difference between the target text length and the average length; Based on the difference and the length standard deviation, text length anomaly information corresponding to the candidate text is generated.
3. The method according to claim 1, wherein, Determining the optimal text length for the candidate text set includes: Obtain an acoustic model evaluation information set for the candidate text set, wherein there is a one-to-one correspondence between the candidate text and the acoustic model evaluation information, and the acoustic model evaluation information is generated based on the acoustic model; The optimal text length is generated based on the acoustic model evaluation information set and the pre-set text length regularization term.
4. The method according to claim 1, wherein, The step of generating the recognized text corresponding to the audio using a pre-trained language model based on the optimal text length and the removed text set includes: From the removed text set, texts whose corresponding text length is the same as the optimal text length are selected to obtain the filtered text set; For each text in the filtered text set, perform the following first generation step: Determine the acoustic model evaluation information and language model evaluation information corresponding to the text, wherein the language model evaluation information is generated based on the language model; Based on the acoustic model evaluation information and the language model evaluation information, generate comprehensive model evaluation information for the text; Texts whose comprehensive evaluation information for the corresponding model meets the first preset evaluation condition are selected from the filtered text set and used as identification texts.
5. The method according to claim 1, wherein, The method further includes: In response to determining that no text length anomaly information representing a text length anomaly exists in the text length anomaly information set, for each candidate text in the candidate text set, the following second generation step is performed: Determine the acoustic model evaluation information and language model evaluation information corresponding to the candidate text; Based on the acoustic model evaluation information and the language model evaluation information, generate comprehensive model evaluation information for the text; Texts whose comprehensive evaluation information for the corresponding model meets the second preset evaluation condition are selected from the candidate text set and used as the identification texts.
6. The method according to claim 4, wherein, The acoustic model evaluation information is the acoustic model evaluation score, and the language model evaluation information is the language model evaluation score; as well as The step of generating comprehensive model evaluation information for the text based on the acoustic model evaluation information and the language model evaluation information includes: The acoustic model evaluation score and the language model evaluation score are weighted and summed to generate a weighted sum score, which serves as the comprehensive evaluation information of the model; and The step of selecting texts from the filtered text set whose corresponding model comprehensive evaluation information meets the first preset evaluation condition as identification texts includes: The text with the highest weighted summation score is selected from the filtered text set and used as the identification text.
7. An audio recognition device, comprising: The first generation unit is configured to generate a candidate text set targeting audio feature information using a pre-trained acoustic model, wherein the target audio feature information corresponds to the audio of the text to be identified. The first determining unit is configured to determine the text length anomaly information corresponding to each candidate text in the candidate text set, thereby obtaining a text length anomaly information set. The removal unit is configured to remove candidate texts that represent text length abnormalities from the candidate text set in response to determining that there are text length abnormality information in the text length abnormality information set, thereby obtaining a removed text set. The second determining unit is configured to determine the optimal text length for the candidate text set; The second generation unit is configured to generate the recognition text corresponding to the audio using a pre-trained language model based on the optimal text length and the removed text set.
8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.
9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.
10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.