An audio data recognition method and apparatus

Abstract

The application discloses a kind of identification method and device of audio data, it is related to computer technical field.The specific embodiment of the method includes: audio data is input to first identification model, and according to the output result of first identification model, determine multiple preferred identification results and optimal confidence score;According to the selection standard that corresponding optimal confidence score is less than preset first threshold value, from audio data, select target audio data input to target identification model, to obtain second identification result by target identification model;Multiple preferred identification results are input into sequence conversion model, and multiple sequence conversion results are obtained;According to multiple sequence conversion results, the second identification result is checked, and in the case where checking passes, the second identification result is taken as the target identification result of audio data.The embodiment can not need to rely on the auxiliary information of third party speech recognition model, improve the identification efficiency and accuracy of audio data, reduce artificial cost.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for recognizing audio data. Background Technology

[0002] Currently, there are two main approaches to audio data recognition: Approach 1 involves manually labeling the audio data to train an acoustic model; Approach 2 involves obtaining recognition results from a high-accuracy third-party speech recognition model, combining this with auxiliary information (such as Nbest scores and confidence levels) to filter the results and obtain highly accurate results as audio annotations for acoustic model training. Here, Nbest represents the N optimal recognition results with the highest confidence levels.

[0003] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art:

[0004] Option 1 has high labor costs and low efficiency in audio data recognition; Option 2 has low recognition accuracy and requires the assistance of third-party speech recognition models. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an audio data recognition method and apparatus that can improve the recognition efficiency and accuracy of audio data and reduce labor costs without relying on auxiliary information from third-party speech recognition models.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for recognizing audio data is provided.

[0007] An audio data recognition method includes: inputting audio data into a first recognition model; determining multiple preferred recognition results and an optimal confidence score based on the output of the first recognition model; selecting target audio data from the audio data and inputting it into a target recognition model according to a selection criterion that the optimal confidence score is less than a preset first threshold, so as to obtain a second recognition result through the target recognition model; inputting the multiple preferred recognition results into a sequence conversion model to obtain multiple sequence conversion results; verifying the second recognition result based on the multiple sequence conversion results; and if the verification passes, using the second recognition result as the target recognition result of the audio data.

[0008] Optionally, the step of inputting audio data into a first recognition model and determining multiple preferred recognition results and optimal confidence scores based on the output of the first recognition model includes: inputting the audio data into the first recognition model to obtain multiple first recognition results and a confidence score for each first recognition result; sorting the confidence scores of each first recognition result and selecting a preset number of first recognition results with the highest confidence scores as the preferred recognition results; and taking the highest confidence score among all confidence scores as the optimal confidence score.

[0009] Optionally, before inputting the multiple preferred recognition results into the sequence conversion model to obtain multiple sequence conversion results, the method includes: calculating the edit distance between the second recognition result and each of the preferred recognition results respectively, averaging all the edit distances to obtain the average edit distance, calculating the ratio of the average edit distance to the string length of the second recognition result; comparing the ratio with a preset second threshold, and determining that the ratio is greater than the preset second threshold.

[0010] Optionally, before inputting the multiple preferred recognition results into the sequence conversion model to obtain multiple sequence conversion results, the method includes: using at least a portion of the audio data as a first audio data sample, labeling the first audio data sample to obtain the labeling result of the first audio data sample; using each preferred recognition result of the first audio data sample as input and the labeling result of the first audio data sample as output to train the sequence conversion model.

[0011] Optionally, the step of verifying the second identification result based on the plurality of sequence conversion results includes: comparing the plurality of sequence conversion results with the second identification result; if the second identification result is the same as any of the sequence conversion results, the verification passes; otherwise, the verification fails.

[0012] Optionally, after using the second recognition result as the target recognition result of the audio data, the method further includes: using the audio data and the target recognition result of the audio data as training data to train the first recognition model, wherein the audio data is used as the input of the first recognition model and the target recognition result of the audio data is used as the output of the first recognition model.

[0013] Optionally, before obtaining the second recognition result through the target recognition model, the method includes: using at least a portion of the audio data as a second audio data sample, labeling the second audio data sample to obtain a labeling result for the second audio data sample; testing the first recognition model and multiple second recognition models using the second audio data sample and the labeling result for the second audio data sample to obtain the recognition rate of the first recognition model and the recognition rate of each second recognition model; and selecting the second recognition model with the highest recognition rate and higher than the recognition rate of the first recognition model from the multiple second recognition models as the target recognition model.

[0014] According to another aspect of the present invention, an audio data recognition device is provided.

[0015] An audio data recognition device includes: a preferred recognition result generation module, used to input audio data into a first recognition model and determine multiple preferred recognition results and an optimal confidence score based on the output of the first recognition model; a second recognition result generation module, used to select target audio data from the audio data and input it into a target recognition model according to a selection criterion that the corresponding optimal confidence score is less than a preset first threshold, so as to obtain a second recognition result through the target recognition model; a sequence conversion result generation module, used to input the multiple preferred recognition results into a sequence conversion model to obtain multiple sequence conversion results; and a target recognition result determination module, used to verify the second recognition result based on the multiple sequence conversion results, and if the verification passes, to use the second recognition result as the target recognition result of the audio data.

[0016] Optionally, the preferred recognition result generation module is further configured to: input the audio data into the first recognition model to obtain multiple first recognition results and a confidence score for each first recognition result; sort the confidence scores of each first recognition result and select a preset number of first recognition results with the highest confidence scores as the preferred recognition results; and take the highest confidence score among all confidence scores as the optimal confidence score.

[0017] Optionally, it further includes a comparison module, configured to: calculate the edit distance between the second recognition result and each of the preferred recognition results respectively; average all edit distances to obtain an average edit distance; calculate the ratio of the average edit distance to the string length of the second recognition result; compare the ratio with a preset second threshold, and determine that the ratio is greater than the preset second threshold.

[0018] Optionally, it further includes a sequence conversion model training module, used to: use at least a portion of the audio data as a first audio data sample, annotate the first audio data sample to obtain the annotation result of the first audio data sample; use each preferred recognition result of the first audio data sample as input and the annotation result of the first audio data sample as output to train the sequence conversion model.

[0019] Optionally, the target recognition result determination module is further configured to: compare the plurality of sequence conversion results with the second recognition result; if the second recognition result is the same as any of the sequence conversion results, the verification passes; otherwise, the verification fails.

[0020] Optionally, it further includes a first recognition model training module, used to: train the first recognition model by using the audio data and the target recognition result of the audio data as training data, wherein the audio data is used as the input of the first recognition model and the target recognition result of the audio data is used as the output of the first recognition model.

[0021] Optionally, it further includes a target recognition model determination module, configured to: use at least a portion of the audio data as a second audio data sample, annotate the second audio data sample to obtain the annotation result of the second audio data sample; test the first recognition model and multiple second recognition models using the second audio data sample and the annotation result of the second audio data sample to obtain the recognition rate of the first recognition model and the recognition rate of each second recognition model; and select the second recognition model with the highest recognition rate and higher than the recognition rate of the first recognition model from the multiple second recognition models as the target recognition model.

[0022] According to another aspect of the present invention, an electronic device is provided.

[0023] An electronic device includes: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the audio data recognition method provided in the embodiments of the present invention.

[0024] According to another aspect of the present invention, a computer-readable medium is provided.

[0025] A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the audio data recognition method provided in the embodiments of the present invention.

[0026] One embodiment of the above invention has the following advantages or beneficial effects: Audio data is input into a first recognition model; multiple preferred recognition results and optimal confidence scores are determined based on the output of the first recognition model; target audio data is selected from the audio data and input into a target recognition model according to the selection criterion that the corresponding optimal confidence score is less than a preset first threshold, so as to obtain a second recognition result through the target recognition model; multiple preferred recognition results are input into a sequence conversion model to obtain multiple sequence conversion results; the second recognition result is verified based on the multiple sequence conversion results; if the verification passes, the second recognition result is used as the target recognition result of the audio data. This approach can improve the recognition efficiency and accuracy of audio data without relying on auxiliary information from third-party speech recognition models, and reduce labor costs.

[0027] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0028] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0029] Figure 1 This is a schematic diagram of the main steps of an audio data recognition method according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the main flow of an audio data recognition method according to an embodiment of the present invention;

[0031] Figure 3 This is a schematic flowchart of an audio data recognition method according to an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the main modules of an audio data recognition device according to an embodiment of the present invention;

[0033] Figure 5 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0034] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0035] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0036] Figure 1 This is a schematic diagram of the main steps of an audio data recognition method according to an embodiment of the present invention.

[0037] like Figure 1 As shown, an embodiment of the audio data recognition method of the present invention mainly includes the following steps S101 to S104.

[0038] Step S101: Input the audio data into the first recognition model, and determine multiple preferred recognition results and the optimal confidence score based on the output of the first recognition model. The first recognition model is a recognition model capable of obtaining the Nbest candidate and confidence score, such as a local recognition model, specifically a local ASR (Automatic Speech Recognition).

[0039] The process of inputting audio data into a first recognition model and determining multiple preferred recognition results and optimal confidence scores based on the output of the first recognition model may include: inputting audio data into the first recognition model to obtain multiple first recognition results and confidence scores for each first recognition result; sorting the confidence scores of each first recognition result and selecting a preset number of first recognition results with the highest confidence scores as preferred recognition results; and taking the highest confidence score among all confidence scores as the optimal confidence score.

[0040] Step S102: Select target audio data from the audio data and input it into the target recognition model according to the selection criteria that the corresponding optimal confidence score is less than the preset first threshold, so as to obtain the second recognition result through the target recognition model.

[0041] Audio data is input into a first recognition model to obtain a first recognition result and a confidence score for the first recognition result. In this embodiment, this confidence score is referred to as the confidence score corresponding to the audio data. If the confidence score is the optimal confidence score, it is also referred to as the optimal confidence score corresponding to the audio data. Target audio data is selected from the audio data and input into the target recognition model according to the selection criterion that the corresponding optimal confidence score is less than a preset first threshold. That is, audio data with corresponding optimal confidence scores and optimal confidence scores less than the preset first threshold are selected as target audio data.

[0042] Before inputting multiple preferred recognition results into the sequence conversion model to obtain multiple sequence conversion results, the following steps may be taken: calculating the edit distance between the second recognition result and each preferred recognition result respectively, averaging all edit distances to obtain the average edit distance, calculating the ratio of the average edit distance to the string length of the second recognition result; comparing the ratio with a preset second threshold, and determining that the ratio is greater than the preset second threshold.

[0043] Before obtaining the second recognition result through the target recognition model, the process may include: using at least a portion of the audio data as second audio data samples, labeling the second audio data samples to obtain the labeling results of the second audio data samples; testing the first recognition model and multiple second recognition models using the second audio data samples and the labeling results of the second audio data samples to obtain the recognition rate of the first recognition model and the recognition rate of each second recognition model; selecting the second recognition model with the highest recognition rate that is higher than the recognition rate of the first recognition model from among the multiple second recognition models as the target recognition model. Here, the second recognition model is a recognition model that cannot obtain the Nbest candidate and confidence score, such as a third-party recognition model. The recognition rate can be 1 minus the word error rate, where the word error rate equals the total number of incorrect words divided by the total number of correctly recognized words.

[0044] Step S103: Input multiple preferred identification results into the sequence conversion model to obtain multiple sequence conversion results.

[0045] Before inputting multiple preferred recognition results into the sequence conversion model to obtain multiple sequence conversion results, the process may include: using at least a portion of the audio data as a first audio data sample, labeling the first audio data sample to obtain the labeling result of the first audio data sample; using each preferred recognition result of the first audio data sample as input and the labeling result of the first audio data sample as output to train the sequence conversion model.

[0046] Sequence transformation models can adopt the transformer model structure (a classic NLP model).

[0047] Step S104: Verify the second recognition result based on multiple sequence conversion results. If the verification passes, use the second recognition result as the target recognition result of the audio data.

[0048] The verification of the second identification result based on multiple sequence conversion results may include: comparing multiple sequence conversion results and the second identification result; if the second identification result is the same as any sequence conversion result, the verification passes; otherwise, the verification fails.

[0049] After using the second recognition result as the target recognition result of the audio data, it may further include: using the audio data and the target recognition result of the audio data as training data to train the first recognition model, wherein the audio data is used as the input of the first recognition model and the target recognition result of the audio data is used as the output of the first recognition model.

[0050] Figure 2 This is a schematic diagram of the main process of an audio data recognition method according to an embodiment of the present invention.

[0051] like Figure 2 As shown, the first step involves a data screening module to identify audio data most likely to improve the local ASR (Automatic Speech Recognition) modeling capabilities. Then, a third-party ASR is used to recognize the pre-screened audio data. The recognition results from the third-party ASR are then processed through a sequence-to-sequence transformation model to select the most reliable results as the labeled audio data, thus obtaining effective training data. The local ASR serves as the first recognition model, and the third-party ASR serves as the second recognition model. This embodiment of the invention can leverage high-quality third-party speech model recognition results without requiring auxiliary information, quickly expanding the effective data for acoustic model training, improving the accuracy of audio data recognition, and reducing labor costs.

[0052] In one embodiment, speech recognition applications are generally related to specific scenarios, such as intelligent customer service, in-vehicle systems, and smart homes. The audio content of each scenario has its own unique bias, and large amounts of audio data can be roughly divided into different categories, i.e., domains, based on the source scenario. The audio content within each domain is related, while there is no clear correlation between different domains. The audio data in this embodiment can be audio data from the same domain.

[0053] In one embodiment, audio data is input into a first recognition model to obtain multiple first recognition results and a confidence score for each first recognition result; the confidence scores of each first recognition result are sorted, and a preset number of first recognition results with the highest confidence scores are selected as preferred recognition results; the highest confidence score among all confidence scores is taken as the optimal confidence score. Specifically, for each audio data, local ASR (i.e., the first recognition model) is used for recognition to obtain multiple local recognition results (i.e., first recognition results) and a confidence score for each local recognition result. N (i.e., a preset number) local recognition results with the highest confidence scores are obtained through Nbest candidates and taken as preferred recognition results, denoted as l_nbest_i (i = 1 to N), and the highest confidence score among all confidence scores is taken as the optimal confidence score.

[0054] In one embodiment, target audio data is selected from the audio data according to the selection criterion that the corresponding optimal confidence score is less than a preset first threshold. Specifically, audio data with a confidence score lower than a certain threshold_1 (i.e., the preset first threshold) is selected as target audio data, along with the corresponding ASR recognition result information, as the initial screening result. Here, the audio recognition model is essentially a statistical modeling of the correspondence between audio and modeling units. Therefore, the statistical modeling capability can be improved by establishing correspondences that are less common or not yet present in the audio recognition model. The confidence score of the audio data recognition result indicates the reliability of the current recognition result. Assuming the confidence score ranges from 0 to 100, the closer to 100, the more reliable; conversely, the closer to 0, the less reliable. If the confidence score of a certain audio data recognition result is low, it means that the relationship between its audio content and the corresponding modeling unit has not been fully established in the current audio recognition model. Adding this audio data to the training data of the audio recognition model can effectively compensate for the missing correspondences in the audio recognition model, improving its robustness and modeling capability. Therefore, the initial screening should select audio data with low confidence scores.

[0055] Figure 3 This is a schematic diagram illustrating the specific process of an audio data recognition method according to an embodiment of the present invention.

[0056] like Figure 3As shown, in one embodiment, target audio data is selected from the audio data and input to the target recognition model to obtain a second recognition result through the target recognition model; the edit distance between the second recognition result and each preferred recognition result is calculated, and the average edit distance is obtained by averaging all the edit distances; the ratio of the average edit distance to the string length of the second recognition result is calculated; the ratio is compared with a preset second threshold, and it is determined that the ratio is greater than the preset second threshold. Specifically, for any target audio data, recognition is performed using a third-party ASR to obtain the recognition result t_rec (i.e., the second recognition result). The edit distance between t_rec and l_nbest_i (i = 1 to N) (i.e., each preferred recognition result) is calculated, and the average edit distance is obtained by averaging the average edit distance by the string length of t_rec to obtain the ratio avg_cer. If avg_cer is greater than the threshold threshold_2 (i.e., the preset second threshold), the process of recognizing the audio data continues; otherwise, the current data of the audio data is discarded, and the next audio data is recognized. The local ASR recognition results for the audio data all have low confidence levels, meaning they are likely to be incorrect. To find potentially correct recognition results using third-party ASR, the local ASR results should differ somewhat from the third-party ASR results. Therefore, the audio data undergoes a second filtering process to find third-party ASR results that differ significantly from the local ASR results. Third-party ASRs can be used by calling their APIs (Application Programming Interfaces).

[0057] In one embodiment, the identification result a = a1...a m And the recognition results b = b1...b n The formula for calculating edit distance is as follows:

[0058]

[0059]

[0060]

[0061] The initialization is performed using formulas (1) and (2), and the iteration is performed continuously using formula (3) to finally obtain d. mn This is the calculated edit distance. When calculating t_rec and l_nbest_i (i = 1 to N), the third-party ASR recognition result can be used as a, then m = 1, and the local ASR recognition result can be used as b to calculate the edit distance between t_rec and each l_nbest_i (i = 1 to N).

[0062] In one embodiment, at least a portion of the audio data is used as a second audio data sample. This second audio data sample is labeled to obtain the labeling results. The first recognition model and multiple second recognition models are tested using the second audio data sample and its labeling results to obtain the recognition rate of the first recognition model and the recognition rate of each second recognition model. The second recognition model with the highest recognition rate, exceeding that of the first recognition model, is selected as the target recognition model. Specifically, a small portion of the initially screened audio data is selected as a test set (i.e., the second audio data sample). The corresponding labeling results are obtained through manual labeling and are considered correct recognition results. The recognition rates of different third-party ASRs (i.e., the second recognition models) for this test set are calculated. The ASR with the highest recognition rate, exceeding that of the local ASR, is selected as the final third-party ASR (i.e., the target recognition model). The recognition rate of the test set is 1 minus the character error rate (WOR). The WRR equals the total number of incorrect characters divided by the total number of correctly recognized characters. The WRR is a commonly used indicator for evaluating ASR accuracy; a lower WRR indicates better recognition performance.

[0063] In one embodiment, multiple preferred recognition results are input into a sequence conversion model to obtain multiple sequence conversion results. These multiple sequence conversion results are compared with a second recognition result. If the second recognition result is the same as any sequence conversion result, the verification passes, and the second recognition result is used as the target recognition result for the audio data. Specifically, local ASR recognition results l_nbest_i (i = 1 to N) that differ significantly from the third-party ASR recognition result t_rec are input into the sequence conversion model to obtain the converted result t_nbest_i (i = 1 to N). The third-party ASR recognition result t_rec is compared with N t_nbest_i strings. If t_rec is the same as any t_nbest_i, t_rec is considered a reliable recognition result and is used as the target recognition result for the audio data. If t_rec is different from any t_nbest_i, the audio data is discarded.

[0064] In one embodiment, at least a portion of the audio data is used as a first audio data sample. The first audio data sample is labeled to obtain a labeling result. The preferred recognition results of the first audio data sample are used as inputs, and the labeling results of the first audio data sample are used as outputs to train the sequence conversion model. Specifically, a portion of the initially screened audio data is selected as the data sample for training the sequence conversion model (i.e., the first audio data sample), and the labeling results of the data sample are obtained through manual labeling. The sequence conversion model can adopt a transformer model structure (a classic NLP model), using each local ASR recognition result as input to the sequence conversion model and the corresponding labeling results as outputs to train the sequence conversion model.

[0065] In one embodiment, audio data and the target recognition results of the audio data are used as training data to train a first recognition model, wherein the audio data serves as the input to the first recognition model, and the target recognition results of the audio data serve as the output of the first recognition model. Specifically, the audio data from which the target recognition results are obtained can be considered effective data for improving the local ASR recognition capability. The audio data and the target recognition results of the audio data are used as training data to train the local ASR. A portion of the training data can be selected as a test set to test the change in the recognition rate of the local ASR before and after training using the training data, in order to verify the effectiveness of the training data. In one embodiment, the training data can also be used to iteratively train the sequence conversion model, continuously improving the accuracy of the sequence conversion model.

[0066] Figure 4 This is a schematic diagram of the main modules of an audio data recognition device according to an embodiment of the present invention.

[0067] like Figure 4 As shown, an audio data recognition device 400 according to an embodiment of the present invention mainly includes: a preferred recognition result generation module 401, a second recognition result generation module 402, a sequence conversion result generation module 403, and a target recognition result determination module 404.

[0068] The preferred recognition result generation module 401 is used to input audio data into the first recognition model and determine multiple preferred recognition results and the optimal confidence score based on the output of the first recognition model.

[0069] The second recognition result generation module 402 is used to select target audio data from the audio data and input it into the target recognition model according to the selection criteria that the corresponding optimal confidence score is less than a preset first threshold, so as to obtain the second recognition result through the target recognition model.

[0070] The sequence conversion result generation module 403 is used to input multiple preferred recognition results into the sequence conversion model to obtain multiple sequence conversion results.

[0071] The target recognition result determination module 404 is used to verify the second recognition result based on multiple sequence conversion results. If the verification passes, the second recognition result is used as the target recognition result of the audio data.

[0072] In one embodiment, the preferred recognition result generation module 401 is specifically used to: input audio data into a first recognition model to obtain multiple first recognition results and a confidence score for each first recognition result; sort the confidence scores of each first recognition result and select a preset number of first recognition results with the highest confidence scores as preferred recognition results; and take the highest confidence score among all confidence scores as the optimal confidence score.

[0073] In one embodiment, a comparison module may be further included, configured to: calculate the edit distance between the second recognition result and each preferred recognition result respectively; average all edit distances to obtain the average edit distance; calculate the ratio of the average edit distance to the string length of the second recognition result; compare the ratio with a preset second threshold; and determine that the ratio is greater than the preset second threshold.

[0074] In one embodiment, the system may further include a sequence conversion model training module, configured to: use at least a portion of the audio data as a first audio data sample, annotate the first audio data sample to obtain the annotation result of the first audio data sample; and use each preferred recognition result of the first audio data sample as input and the annotation result of the first audio data sample as output to train the sequence conversion model.

[0075] In one embodiment, the target recognition result determination module 404 is specifically used to: compare multiple sequence conversion results and a second recognition result; if the second recognition result is the same as any sequence conversion result, the verification passes; otherwise, the verification fails.

[0076] In one embodiment, a first recognition model training module may be included, configured to: train a first recognition model using audio data and the target recognition result of the audio data as training data, wherein the audio data is used as the input of the first recognition model and the target recognition result of the audio data is used as the output of the first recognition model.

[0077] In one embodiment, the system may further include a target recognition model determination module, configured to: label the second audio data samples using at least a portion of the audio data as second audio data samples to obtain labeling results for the second audio data samples; test the first recognition model and multiple second recognition models using the second audio data samples and the labeling results for the second audio data samples to obtain the recognition rate of the first recognition model and the recognition rate of each second recognition model; and select the second recognition model with the highest recognition rate and a recognition rate higher than that of the first recognition model from among the multiple second recognition models as the target recognition model.

[0078] Furthermore, the specific implementation details of the audio data recognition device in this embodiment of the invention have been described in detail in the audio data recognition method above, so they will not be repeated here.

[0079] Figure 5 An exemplary system architecture 500 is shown, in which the audio data recognition method or audio data recognition device of the present invention can be applied.

[0080] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0081] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0082] Terminal devices 501, 502, and 503 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0083] Server 505 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 501, 502, and 503 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal device.

[0084] It should be noted that the audio data recognition method provided in this embodiment of the invention is generally executed by server 505, and correspondingly, the audio data recognition device is generally set in server 505.

[0085] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0086] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing terminal devices or servers of the present invention. Figure 6 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0087] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0088] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0089] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0090] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a preferred recognition result generation module, a second recognition result generation module, a sequence conversion result generation module, and a target recognition result determination module. The names of these modules do not necessarily limit the module itself; for example, the preferred recognition result generation module can also be described as "a module for inputting audio data into a first recognition model and determining multiple preferred recognition results and an optimal confidence score based on the output of the first recognition model."

[0093] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: inputting audio data into a first recognition model; determining multiple preferred recognition results and an optimal confidence score based on the output of the first recognition model; selecting target audio data from the audio data and inputting it into a target recognition model according to a selection criterion that the corresponding optimal confidence score is less than a preset first threshold, so as to obtain a second recognition result through the target recognition model; inputting the multiple preferred recognition results into a sequence conversion model to obtain multiple sequence conversion results; verifying the second recognition result based on the multiple sequence conversion results; and, if the verification passes, using the second recognition result as the target recognition result of the audio data.

[0094] According to the technical solution of this invention, audio data is input into a first recognition model, and multiple preferred recognition results and optimal confidence scores are determined based on the output of the first recognition model. Target audio data is selected from the audio data and input into a target recognition model according to the selection criterion that the corresponding optimal confidence score is less than a preset first threshold, so as to obtain a second recognition result through the target recognition model. The multiple preferred recognition results are input into a sequence conversion model to obtain multiple sequence conversion results. The second recognition result is verified based on the multiple sequence conversion results, and if the verification passes, the second recognition result is used as the target recognition result of the audio data. This method can improve the recognition efficiency and accuracy of audio data without relying on auxiliary information from third-party speech recognition models, and reduce labor costs.

[0095] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for recognizing audio data, characterized in that, include: The audio data is input into the first recognition model, and multiple preferred recognition results and the optimal confidence score are determined based on the output of the first recognition model. According to the selection criterion that the optimal confidence score is less than a preset first threshold, target audio data is selected from the audio data and input into the target recognition model so as to obtain a second recognition result through the target recognition model; The multiple preferred identification results are input into the sequence conversion model to obtain multiple sequence conversion results; The second recognition result is verified based on the multiple sequence conversion results. If the verification passes, the second recognition result is used as the target recognition result of the audio data. Before inputting the multiple preferred recognition results into the sequence conversion model to obtain multiple sequence conversion results, the method includes: calculating the edit distance between the second recognition result and each of the preferred recognition results respectively, averaging all the edit distances to obtain the average edit distance, and calculating the ratio of the average edit distance to the string length of the second recognition result; The ratio is compared with a preset second threshold, and it is determined that the ratio is greater than the preset second threshold.

2. The method according to claim 1, characterized in that, The step of inputting audio data into a first recognition model and determining multiple preferred recognition results and optimal confidence scores based on the output of the first recognition model includes: The audio data is input into the first recognition model to obtain multiple first recognition results and confidence scores for each first recognition result; The confidence scores of each of the first identification results are sorted, and a preset number of the first identification results with the highest confidence scores are selected as the preferred identification results; The highest confidence score among all confidence scores is taken as the optimal confidence score.

3. The method according to claim 1, characterized in that, Before inputting the multiple preferred identification results into the sequence conversion model to obtain multiple sequence conversion results, the process includes: Using at least a portion of the audio data as a first audio data sample, the first audio data sample is labeled to obtain the labeling result of the first audio data sample; The sequence conversion model is trained by taking the preferred recognition results of the first audio data sample as input and the annotation results of the first audio data sample as output.

4. The method according to claim 1, characterized in that, The step of verifying the second identification result based on the multiple sequence conversion results includes: The verification is passed if the second identification result is the same as any of the sequence conversion results; otherwise, the verification fails.

5. The method according to claim 1, characterized in that, After using the second recognition result as the target recognition result of the audio data, the method further includes: The audio data and the target recognition result of the audio data are used as training data to train the first recognition model, wherein the audio data is used as the input of the first recognition model and the target recognition result of the audio data is used as the output of the first recognition model.

6. The method according to claim 1, characterized in that, Before obtaining the second recognition result through the target recognition model, the process includes: Using at least a portion of the audio data as a second audio data sample, the second audio data sample is labeled to obtain the labeling result of the second audio data sample; The first recognition model and multiple second recognition models are tested using the second audio data sample and the annotation results of the second audio data sample to obtain the recognition rate of the first recognition model and the recognition rate of each second recognition model. The second recognition model with the highest recognition rate and higher than the recognition rate of the first recognition model is selected from the plurality of second recognition models as the target recognition model.

7. An audio data recognition device, characterized in that, include: The preferred recognition result generation module is used to input audio data into the first recognition model and determine multiple preferred recognition results and the optimal confidence score based on the output of the first recognition model. The second recognition result generation module is used to select target audio data from the audio data and input it into the target recognition model according to the selection criterion that the corresponding optimal confidence score is less than a preset first threshold, so as to obtain the second recognition result through the target recognition model; The sequence conversion result generation module is used to input the multiple preferred identification results into the sequence conversion model to obtain multiple sequence conversion results; The target recognition result determination module is used to verify the second recognition result based on the multiple sequence conversion results, and if the verification passes, to use the second recognition result as the target recognition result of the audio data; The device further includes a comparison module, configured to: calculate the edit distance between the second recognition result and each of the preferred recognition results respectively, average all edit distances to obtain the average edit distance, and calculate the ratio of the average edit distance to the string length of the second recognition result; The ratio is compared with a preset second threshold, and it is determined that the ratio is greater than the preset second threshold.

8. The apparatus according to claim 7, characterized in that, The preferred identification result generation module is also used for: The audio data is input into the first recognition model to obtain multiple first recognition results and confidence scores for each first recognition result; The confidence scores of each of the first identification results are sorted, and a preset number of the first identification results with the highest confidence scores are selected as the preferred identification results; The highest confidence score among all confidence scores is taken as the optimal confidence score.

9. The apparatus according to claim 7, characterized in that, It also includes a sequence transformation model training module, used for: Using at least a portion of the audio data as a first audio data sample, the first audio data sample is labeled to obtain the labeling result of the first audio data sample; The sequence conversion model is trained by taking the preferred recognition results of the first audio data sample as input and the annotation results of the first audio data sample as output.

10. The apparatus according to claim 7, characterized in that, The target recognition result determination module is also used for: The verification is passed if the second identification result is the same as any of the sequence conversion results; otherwise, the verification fails.

11. The apparatus according to claim 7, characterized in that, It also includes a first recognition model training module, used for: The audio data and the target recognition result of the audio data are used as training data to train the first recognition model, wherein the audio data is used as the input of the first recognition model and the target recognition result of the audio data is used as the output of the first recognition model.

12. The apparatus according to claim 7, characterized in that, It also includes a target recognition model determination module, used for: Using at least a portion of the audio data as a second audio data sample, the second audio data sample is labeled to obtain the labeling result of the second audio data sample; The first recognition model and multiple second recognition models are tested using the second audio data sample and the annotation results of the second audio data sample to obtain the recognition rate of the first recognition model and the recognition rate of each second recognition model. The second recognition model with the highest recognition rate and higher than the recognition rate of the first recognition model is selected from the plurality of second recognition models as the target recognition model.

13. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

14. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images