Audio data processing method and audio processing device

Abstract

The present application discloses an audio data processing method in the field of artificial intelligence. The method comprises: obtaining audio data to be processed; inputting the audio data to be processed into a first automatic speech recognition model, and when the phoneme probability matrix output by the first automatic speech recognition model includes a dictionary probability vector of a non-blank frame and a dictionary probability vector of a blank frame, removing the dictionary probability vector of the blank frame in the phoneme probability matrix to obtain a valid phoneme probability matrix; obtaining a target phoneme sequence according to the valid phoneme probability matrix; inputting the audio data to be processed into a second automatic speech recognition model; determining a reference phoneme sequence according to a reference text sequence output by the second automatic speech recognition model; and when the target phoneme sequence is aligned with the reference phoneme sequence, determining that the label of the audio data to be processed is a reference text sequence. The method can automatically annotate audio data with high accuracy. The present application also provides an audio processing device capable of implementing the above method.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an audio data processing method and an audio processing device. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] With the development of artificial intelligence technology, people can speak their intentions to electronic products such as mobile phones, speakers, and TVs, and electronic products can perform corresponding functions based on the user's voice. With the rise of voice recognition technology, users' requirements for its performance are gradually increasing, and the most urgent demand is the need for high voice recognition accuracy.

[0004] An existing speech annotation method is roughly as follows: receiving speech data, performing speech recognition on the speech data, and obtaining recognized text; obtaining user confirmation text after the user confirms the recognized text; analyzing the recognized text and the user confirmation text, and extracting automatic annotation features; annotating the speech data according to the automatic annotation features and a pre-built automatic annotation model.

[0005] In this method, when using the speech recognition service, the user needs to take additional actions to provide confirmation information for the recognition result, and the additional confirmation action seriously affects the user's experience of using the speech recognition service. This method relies on the user to confirm the text, and the speech annotation efficiency is not high. Summary of the invention

[0006] In view of this, the present application provides an audio data processing method and device, which can automatically annotate audio data to improve annotation efficiency.

[0007] The first aspect provides an audio data processing method, in which the audio data to be processed is obtained; the audio data to be processed is input into a first automatic speech recognition model, and the first automatic speech recognition model outputs a phoneme probability matrix according to the audio data to be processed; when the phoneme probability matrix includes a dictionary probability vector of a non-blank frame and a dictionary probability vector of a blank frame, the dictionary probability vector of the blank frame in the phoneme probability matrix is ​​removed to obtain a valid phoneme probability matrix; a target phoneme sequence is obtained according to the valid phoneme probability matrix; the audio data to be processed is input into a second automatic speech recognition model; a reference phoneme sequence is determined according to a reference text sequence output by the second automatic speech recognition model; when the target phoneme sequence is aligned with the reference phoneme sequence, the label of the audio data to be processed is determined to be the reference text sequence. The audio data to be processed includes multiple speech frames, and the phoneme probability matrix includes a dictionary probability vector for each speech frame.

[0008] According to this implementation, the target phoneme sequence and the reference phoneme sequence can be obtained respectively according to the output results of the two automatic speech recognition models. When the target phoneme sequence and the reference phoneme sequence are aligned, it indicates that the target phoneme sequence and the reference phoneme sequence are the same, and the label of the audio data to be processed is determined to be the reference text sequence, so that the audio data can be automatically annotated, thereby improving the annotation efficiency. When the user confirms the annotation method, the user may have some incorrect annotations, and the present application compares the phoneme sequence to determine whether the annotation is correct, which has higher accuracy.

[0009] In a possible implementation, when the target phoneme sequence is not aligned with the reference phoneme sequence, the proportion of uneven phonemes in the target phoneme sequence is determined; the phoneme confusion vector is determined based on the valid phoneme probability matrix; the confusion of all uneven phonemes is selected from the phoneme confusion vector; when the proportion of uneven phonemes is less than or equal to a preset proportion and the confusion of all uneven phonemes is less than the preset confusion, the label of the audio data to be processed is determined to be the text sequence corresponding to the target phoneme sequence. When the proportion of uneven phonemes is less than or equal to a preset proportion and the confusion of all uneven phonemes is less than the preset confusion, it indicates that only a small number of phonemes are different between the two phoneme sequences, and the credibility of the text corresponding to the uneven phonemes is high, so that the text sequence corresponding to the target phoneme sequence has a high credibility.

[0010] In another possible implementation, determining the phoneme confusion degree vector according to the effective phoneme probability matrix includes: determining the confusion degree of the speech frame according to the dictionary probability vector of the speech frame in the effective phoneme probability matrix; and forming the phoneme confusion degree vector from the confusion degree of each speech frame according to the order of the dictionary probability vector of the speech frame in the effective phoneme probability matrix. The order of the dictionary probability vector of the speech frame is consistent with the order of the speech frame.

[0011] In another possible implementation, determining the confusion degree of a speech frame according to the dictionary probability vector of the speech frame in the effective phoneme probability matrix includes: arranging all dictionary probabilities in the dictionary probability vector of the speech frame in descending order; for the first k dictionary probabilities in the dictionary probability vector after descending order, selecting a confusing dictionary probability set from the first k dictionary probabilities, the confusing dictionary probability set including all dictionary probabilities greater than a preset probability, k being a positive integer; determining a confusion ratio according to the total number of dictionary probabilities included in the confusing dictionary probability set and k; determining a maximum probability gradient according to the maximum dictionary probability and the minimum dictionary probability among the first k dictionary probabilities; and determining the confusion degree of the speech frame according to the confusion ratio and the maximum probability gradient. This provides a method for calculating the confusion degree of a speech frame.

[0012] In another possible implementation, when the target phoneme sequence is aligned with the reference phoneme sequence, the type of the audio data to be processed is determined to be standard audio.

[0013] In another possible implementation, when the target phoneme sequence is not aligned with the reference phoneme sequence, the proportion of uneven phonemes in the target phoneme sequence is determined; when the proportion of uneven phonemes in the target phoneme sequence is greater than a preset ratio, the type of the audio data to be processed is determined to be noise.

[0014] In another possible implementation, when the proportion of uneven phonemes in the target phoneme sequence is less than or equal to a preset proportion, a phoneme confusion vector is determined based on a valid phoneme probability matrix; the confusion degrees of all uneven phonemes are selected from the phoneme confusion vector; when the confusion degrees of all uneven phonemes are less than the preset confusion degree, the type of the audio data to be processed is determined to be standard audio; when the confusion degree of at least one uneven phoneme is greater than or equal to the preset confusion degree, the type of the audio data to be processed is determined to be partially word blurred audio.

[0015] In another possible implementation, a first valid phoneme position sequence is obtained based on a valid phoneme probability matrix; when the first valid phoneme position sequence contains at least one valid phoneme position, the first valid phoneme position sequence is shifted; an average value is calculated based on the difference between a second valid phoneme position sequence obtained by the shift and the first valid phoneme position sequence; when the average value is greater than a preset threshold, it is determined that the type of the audio signal to be processed is fast-speech audio.

[0016] In another possible implementation, when the first valid grapheme position sequence does not include a valid grapheme position, the audio data to be processed is input into a speech activation detection model, and the speech activation detection model outputs a voicing interval according to the audio data to be processed; when the number of voicing intervals is greater than a preset number of intervals, the type of the audio data to be processed is determined to be noise; when the number of voicing intervals is less than or equal to a preset number of intervals, the audio data to be processed is discarded. The preset number of intervals may be, but is not limited to, 2.

[0017] In the above implementation, the first automatic speech recognition model, the second automatic speech recognition model and the speech activation detection model are all deep neural network models.

[0018] In another possible implementation, when the number of sampling points of the audio data to be processed does not belong to a preset interval, it is determined that the audio data to be processed is invalid audio data.

[0019] The second aspect provides an audio processing device, which includes a first acquisition unit, a first speech recognition unit, a removal unit, a second acquisition unit, a second speech recognition unit, a third acquisition unit and a label unit; the first acquisition unit is used to acquire audio data to be processed, and the audio data to be processed includes multiple speech frames; the first speech recognition unit is used to input the audio data to be processed into a first automatic speech recognition model, and the first automatic speech recognition model is used to output a phoneme probability matrix according to the audio data to be processed, and the phoneme probability matrix includes dictionary probability vectors of multiple speech frames; the removal unit is used when the phoneme probability matrix includes dictionary probability vectors of non-blank frames When the probability vector and the dictionary probability vector of the blank frame are obtained, the dictionary probability vector of the blank frame in the phoneme probability matrix is ​​removed to obtain a valid phoneme probability matrix; a second acquisition unit is used to obtain a target phoneme sequence according to the valid phoneme probability matrix; a second speech recognition unit is used to input the audio data to be processed into a second automatic speech recognition model, and the first automatic speech recognition model is different from the second automatic speech recognition model; a third acquisition unit determines a reference phoneme sequence according to a reference text sequence output by the second automatic speech recognition model; a label unit is used to determine that the label of the audio data to be processed is the reference text sequence when the target phoneme sequence is aligned with the reference phoneme sequence.

[0020] In a possible implementation, the audio processing device further includes: a first determination unit is further used to determine the proportion of uneven phonemes in the target phoneme sequence; a second determination unit is used to determine the phoneme confusion vector according to the valid phoneme probability matrix; the confusion of all uneven phonemes is selected from the phoneme confusion vector; the label unit is further used to determine that the label of the audio data to be processed is the text sequence corresponding to the target phoneme sequence when the proportion of uneven phonemes is less than or equal to a preset ratio and the confusion of all uneven phonemes is less than the preset confusion. Wherein, the target phoneme sequence is not aligned with the reference phoneme sequence.

[0021] In another possible implementation, the second determining unit includes a determining subunit and a composing subunit; the determining subunit is used to determine the confusion degree of the speech frame according to the dictionary probability vector of the speech frame in the effective phoneme probability matrix; the composing subunit is used to compose the confusion degree of each speech frame into a phoneme confusion degree vector according to the order of the dictionary probability vector in the effective phoneme probability matrix. Based on the phoneme confusion degree vector, it can be determined which words in the audio data to be processed are pronounced ambiguously.

[0022] In another possible implementation, the determination subunit is specifically used to arrange all dictionary probabilities in the dictionary probability vector of the speech frame in descending order; for the first k dictionary probabilities in the dictionary probability vector after the descending order, select a confusing dictionary probability set from the first k dictionary probabilities, the confusing dictionary probability set includes all dictionary probabilities greater than a preset probability, k is a positive integer; determine the confusion ratio based on the total number of dictionary probabilities included in the confusing dictionary probability set and k; determine the maximum probability gradient based on the maximum dictionary probability and the minimum dictionary probability among the first k dictionary probabilities; and determine the confusion degree of the speech frame based on the confusion ratio and the maximum probability gradient.

[0023] In another possible implementation, the audio processing device further includes a first classification unit, which is used to determine that the type of the audio data to be processed is standard audio when the target phoneme sequence is aligned with the reference phoneme sequence.

[0024] In another possible implementation, the audio processing device further includes a second classification unit, which is used to determine the proportion of uneven phonemes in the target phoneme sequence; when the proportion of uneven phonemes in the target phoneme sequence is greater than a preset proportion, the type of the audio data to be processed is determined to be noise. The target phoneme sequence is not aligned with the reference phoneme sequence.

[0025] In another possible implementation, the audio processing device further includes a third classification unit, which is used to determine a phoneme confusion degree vector according to a valid phoneme probability matrix; select the confusion degrees of all uneven phonemes from the phoneme confusion degree vector; when the confusion degrees of all uneven phonemes are less than a preset confusion degree, determine that the type of the audio data to be processed is standard audio; when the confusion degree of at least one uneven phoneme is greater than or equal to the preset confusion degree, determine that the type of the audio data to be processed is partial word fuzzy audio. The proportion of uneven phonemes in the target phoneme sequence is less than or equal to a preset proportion.

[0026] In another possible implementation, the audio processing device also includes a fourth classification unit, which is used to obtain a first valid phoneme position sequence according to a valid phoneme probability matrix; when the first valid phoneme position sequence contains at least one valid phoneme position, the first valid phoneme position sequence is shifted; an average value is calculated based on the difference between the second valid phoneme position sequence obtained by the shift and the first valid phoneme position sequence; when the average value is greater than a preset threshold, it is determined that the type of the audio signal to be processed is fast-speech audio.

[0027] In another possible implementation, the audio processing device further includes a fifth classification unit, the fifth classification unit is used to input the audio data to be processed into a speech activation detection model, and the speech activation detection model is used to output a vocalization interval according to the audio data to be processed; when the number of vocalization intervals is greater than the preset number of intervals, the type of the audio data to be processed is determined to be noise; when the number of vocalization intervals is less than or equal to the preset number of intervals, the audio data to be processed is discarded. The first valid grapheme phoneme position sequence does not include a valid grapheme phoneme position;

[0028] In another possible implementation, the first automatic speech recognition model, the second automatic speech recognition model, and the speech activation detection model are all deep neural network models.

[0029] In another possible implementation manner, the first acquisition unit is further configured to determine that the audio data to be processed is invalid audio data when the number of sampling points of the audio data to be processed does not belong to a preset interval.

[0030] The information interaction, execution process, etc. between the units / sub-units in the audio processing device of the second aspect and the technical effects brought about by them can all be referred to the corresponding description of the first aspect.

[0031] The third aspect provides a training method, in which a neural network algorithm is used to train first audio data and second audio data to obtain an automatic speech recognition model, wherein the first audio data is standard audio data, and the second audio data includes at least one of noise data, fast speech audio data, and partially ambiguous audio data. According to this implementation, different types of audio data are trained to improve the generalization ability of the automatic speech recognition model, thereby improving the accuracy of audio data recognized by the automatic speech recognition model.

[0032] The fourth aspect provides a training device including a training unit, which is used to use a neural network algorithm to train first audio data and second audio data to obtain an automatic speech recognition model, the first audio data is standard audio data, and the second audio data includes at least one of noise data, fast-speech audio data, and partially blurred audio data.

[0033] The fifth aspect provides an audio processing device, which includes a processor and a memory, the memory is used to store programs and data; the processor is used to implement the audio data processing method of the first aspect by executing the program.

[0034] A sixth aspect provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the methods of the above aspects.

[0035] A seventh aspect provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the methods of the above aspects.

[0036] In an eighth aspect, a chip system is provided, which includes a processor for supporting a corresponding device to implement the functions involved in the above aspects, such as receiving, sending or processing the data involved in the above methods. In a possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the corresponding device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 A schematic diagram of an artificial intelligence main framework of an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a speech recognition system according to an embodiment of the present application;

[0040] Figure 3 A flow chart of the audio data processing method according to an embodiment of the present application;

[0041] Figure 4 Another schematic diagram of a speech recognition system according to an embodiment of the present application;

[0042] Figure 5 is a structural diagram of an audio processing device according to an embodiment of the present application;

[0043] Figure 6 Another structural diagram of the server of an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] First, some terms of this application are introduced:

[0046] A phoneme is the smallest speech unit divided according to the natural attributes of speech. In Chinese speech recognition applications, a phoneme can be understood as the pinyin representation of pronunciation. For example, the character "中" corresponds to "zhong1", and "zhong1" can be regarded as a valid phoneme or a sequence of phonemes. "zhong1" consists of 6 phonemes. Each phoneme includes multiple speech frames, and a speech frame is the information contained within a frame. Another example is that the phoneme sequence corresponding to "中国人" is "zhong1 guo2 ren2".

[0047] If the speech frame recognition result corresponding to a speech frame is empty, then the speech frame is a blank frame. If a speech frame corresponds to one or more speech frame recognition results, then the speech frame is a non - blank frame, also known as a valid frame. The phoneme probability can be determined based on the correspondence between phonemes and speech frames and the recognition results of speech frames.

[0048] Phoneme probability is the probability corresponding to the phoneme and its recognition result. The value of phoneme probability can reflect the degree of confirmation of the acoustic model for this phoneme. For example, the phoneme probability of the acoustic model for "中" is {"zhong1":0.912, "zong3":0.088}.

[0049] The phoneme probability matrix includes the dictionary probability vectors of multiple speech frames. Each word is a matrix obtained by stacking the phoneme probabilities of each frame of the audio output by the acoustic model. The phoneme probability matrix is a two - dimensional matrix, where one dimension is time, with the time unit being frames, and the other dimension is the vocabulary length. Usually, the vocabulary length is pre - designed, and its value can be, but is not limited to, 1598.

[0050] For example, the phoneme probability matrix pw[T, V] is as follows:

[0051]

[0052] T is the total number of speech frames, and V is the vocabulary length.

[0053] is the dictionary probability vector of the first speech frame. The dictionary probability vectors of other speech frames can be deduced by analogy.

[0054] The valid phoneme probability matrix is a matrix composed of the dictionary probability vectors of all remaining valid frames after removing the dictionary probability vectors of blank frames from the original phoneme probability matrix. In the valid phoneme probability matrix, the order of the dictionary probability vectors of valid frames is the same as the order of valid frames.

[0055] The valid phoneme position sequence includes the positions of multiple valid phonemes. The position of a valid phoneme refers to the position sequence of the multiple speech frames included in this valid phoneme.

[0056] A neural network can be understood as a computing model that consists of a large number of nodes (or neurons) connected to each other. Each node represents a specific output function, called an activation function. The connection between each two nodes represents a weighted value for the signal passing through the connection, called a weight, which is equivalent to the memory of the neural network. When the connection mode, weight value and activation function of the neural network change, the output of the network also changes. A neural network can be an approximation of a certain algorithm or function in nature, or it can be an expression of a logical strategy.

[0057] The work of each layer in a deep neural network can be expressed mathematically as To describe: From a physical level, the work of each layer in a deep neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of the neural network. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a deep neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0058] Because we want the output of the deep neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the network's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0059] Figure 1 A schematic diagram of an artificial intelligence main framework is shown, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field requirements.

[0060] The following is an explanation of the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0061] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom".

[0062] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0063] (1) Infrastructure:

[0064] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0065] (2) Data

[0066] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0067] (3) Data processing

[0068] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0069] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0070] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0071] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0072] (4) General ability

[0073] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0074] (5) Smart products and industry applications

[0075] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0076] Figure 2 A schematic diagram of the speech recognition system of the present application. Figure 2 The data acquisition device 260 is used to collect audio data (such as standard audio data) and store it in the database 230. The training device 220 generates a first automatic speech recognition model 201 and a second automatic speech recognition model 202 based on the audio data in the database 230.

[0077] The following will describe in more detail how the training device 220 obtains the first automatic speech recognition model 201 and the second automatic speech recognition model 202 based on the audio data. The first automatic speech recognition model 201 can output a phoneme probability matrix according to the input audio data, and the second automatic speech recognition model 202 can output a text sequence according to the input audio data. It should be understood that the audio data used to train the first automatic speech recognition model 201 and the audio data used to train the second automatic speech recognition model 202 can be different data from the database 230. Alternatively, the neural network algorithm used to train the first automatic speech recognition model 201 and the neural network algorithm used to train the second automatic speech recognition model 202 can be different neural network algorithms. Alternatively, the neural network algorithm used to train the first automatic speech recognition model 201 is the same as the neural network algorithm used to train the second automatic speech recognition model 202, but the network layers of the two automatic speech recognition models are different or other model parameters are different.

[0078] The first automatic speech recognition model 201 and the second automatic speech recognition model 202 can be applied to different systems or devices. Figure 2 In the embodiment, the execution device 210 is configured with an I / O interface 212 to perform data interaction with an external device, and a user can input data to the I / O interface 212 through a user device 240. The I / O interface is an input / output interface.

[0079] The execution device 210 can call data, codes, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .

[0080] The calculation module 211 uses the first automatic speech recognition model 201 to process the input audio data, and obtains a phoneme probability matrix after processing by the first automatic speech recognition model 201. When the phoneme probability matrix includes dictionary probability vectors of non-blank frames and dictionary probability vectors of blank frames, the dictionary probability vectors of blank frames in the phoneme probability matrix are removed to obtain a valid phoneme probability matrix; a target phoneme sequence is obtained according to the valid phoneme probability matrix; the input audio data is processed by the second automatic speech recognition model 202, and a reference text sequence is obtained after processing by the second automatic speech recognition model 202. When the target phoneme sequence is aligned with the reference phoneme sequence, the label of the audio data is determined to be the reference text sequence. When the target phoneme sequence is not aligned with the reference phoneme sequence, determine the proportion of uneven phonemes in the target phoneme sequence; determine the phoneme confusion vector according to the valid phoneme probability matrix; select the confusion of all uneven phonemes from the phoneme confusion vector; when the proportion of uneven phonemes is less than or equal to the preset proportion and the confusion of all uneven phonemes is less than the preset confusion, determine that the label of the audio data to be processed is the text sequence corresponding to the target phoneme sequence. It should be understood that the calculation module 211 can also be used to perform Figure 3An audio data processing method in an alternative embodiment of the illustrated embodiment.

[0081] Finally, the I / O interface 212 returns the label of the audio data of the calculation module 211 to the user device 240 and provides it to the user.

[0082] exist Figure 2 In the illustrated case, the user can manually specify the data input into the execution device 210, for example, by operating in the interface provided by the I / O interface 212. In another case, the user device 240 can automatically input data into the I / O interface 212 and obtain the result. If the user device 240 needs to obtain the user's authorization to automatically input data, the user can set the corresponding authority in the user device 240. The user can view the result output by the execution device 210 on the user device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The user device 240 can also serve as a data acquisition terminal to store the collected audio signal in the database 230.

[0083] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture of an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 2 The data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0084] It should be understood that the first automatic speech recognition model 201 and the second automatic speech recognition model 202 can be trained based on audio data from different databases, or trained by different training devices. The first automatic speech recognition model 201 and the second automatic speech recognition model 202 can be deployed in different execution devices or in different computing units of the same execution device.

[0085] The user device 220 is an electronic device with audio processing capabilities, such as a microphone or a user device with a microphone, etc. The user device 220 can be a personal computer, a tablet computer, a mobile phone, a car computer, a wearable device, a virtual reality device (VR), an augmented reality device (AR), etc. The user device 220 is also called a terminal, a user terminal, or a client device, etc.

[0086] The above is an introduction to the speech recognition system of the present application. The following is an introduction to the audio data processing method of the present application.

[0087] See also Figure 3 , an embodiment of the audio data processing method provided by the present application includes:

[0088] Step 301: Obtain audio data to be processed.

[0089] The audio data to be processed includes multiple speech frames. Optionally, the number of sampling points of the audio data to be processed is obtained. When the number of sampling points of the audio data to be processed does not belong to the preset interval, the audio data to be processed is determined to be invalid audio data, and the following steps are not performed. When the number of sampling points of the audio data to be processed does not belong to the preset interval, the audio data to be processed is determined to be valid audio data, and step 302 is performed. The preset interval can be but is not limited to [8000, 16000]. The upper limit and the lower limit of the interval can be adjusted according to actual conditions, and this application does not limit it.

[0090] Step 302: input the audio data to be processed into a first automatic speech recognition model, and the first automatic speech recognition model is used to output a phoneme probability matrix according to the audio data to be processed.

[0091] The phoneme probability matrix includes dictionary probability vectors for multiple speech frames.

[0092] Step 303: When the phoneme probability matrix includes the dictionary probability vectors of non-blank frames and the dictionary probability vectors of blank frames, the dictionary probability vectors of blank frames in the phoneme probability matrix are removed to obtain a valid phoneme probability matrix.

[0093] The phoneme probability matrix can be recorded as pw[T, V], where T is the total number of speech frames of the audio data to be processed, and V is the word list length, which is equal to the total number of words in the word list. The phoneme probability matrix includes dictionary probability vectors of T speech frames, and the dictionary probability vector of each speech frame includes V dictionary probabilities.

[0094] Specifically, select the dictionary probability vector of the t-th speech frame in the phoneme probability matrix, count the elements whose dictionary probability is 0 and the elements whose dictionary probability is not equal to 0, and determine the probability p[t, 0], t∈[0, T] that the dictionary probability vector of the t-th speech frame is 0 according to the ratio of the number of elements whose dictionary probability is 0 to V. When |p[t, 0]-BT| is greater than 0.5, it is determined that the t-th speech frame is a blank frame. When |p[t, 0]-BT| is less than or equal to 0.5, it is determined that the t-th speech frame is a non-blank frame, that is, a valid frame. BT can be but is not limited to 0.45. The size of BT can be set according to actual conditions and is not limited here.

[0095] After judging each speech frame according to the above method, the dictionary probability vectors of all valid frames can be obtained by removing the dictionary probability vectors of blank frames. According to the order of speech frames, the valid phoneme probability matrix can be recorded as pw[B, V], where B is the total number of valid frames obtained after removing blank frames from all speech frames, and is also the total number of dictionary probability vectors of valid frames.

[0096] Step 304: Obtain a target phoneme sequence according to the valid phoneme probability matrix.

[0097] The method of obtaining the target phoneme sequence according to the valid phoneme probability matrix is ​​as follows: performing a beam search on the valid phoneme probability matrix to obtain the target phoneme sequence. The beam search is also called cluster search.

[0098] Step 305: Input the audio data to be processed into a second automatic speech recognition model.

[0099] The first automatic speech recognition model is different from the second automatic speech recognition model. The difference between the first automatic speech recognition model and the second automatic speech recognition model includes but is not limited to the number of network layers, the number of parameters or the amount of training data. When the second automatic speech model has more network layers, more parameters or more training data, the second automatic speech model has stronger data fitting ability, stronger speech adaptability and higher recognition accuracy. For example, the second automatic speech model is an automatic speech model using a hidden Markov model, or a deep learning network model based on a Transformer structure, or a deep learning network model based on a recurrent neural network (RNN)-Transducer structure.

[0100] Step 306: Determine a reference phoneme sequence according to the reference text sequence output by the second automatic speech recognition model.

[0101] The text can be converted into a phoneme sequence by using word segmentation and phonetic notation. For example, the reference text sequence is [today's weather], and the reference phoneme sequence corresponding to the reference text sequence is [jin1 tian1 tian1 qi4].

[0102] It can be seen from steps 302 to 306 that the target phoneme sequence and the reference phoneme sequence can be obtained respectively according to the output results of the two automatic speech recognition models. It should be understood that steps 302 to 304 are the process of obtaining the target phoneme sequence, and steps 305 to 306 are the process of obtaining the reference phoneme sequence, and the two processes have no fixed order. For example, steps 305 to 306 can be performed before steps 302 to 304.

[0103] Step 307: When the target phoneme sequence is aligned with the reference phoneme sequence, the label of the audio data to be processed is determined to be the reference text sequence.

[0104] When the target phoneme sequence and the reference phoneme sequence are aligned, it indicates that the two phoneme sequences are the same, and the label of the audio data to be processed is determined to be the reference text sequence, so that the audio data can be automatically annotated, thereby improving the annotation efficiency. According to the manual audio annotation method, a person can annotate 800-1000 audio data per day, and according to the method of this embodiment, about 70 million audio data can be annotated every day. When the user confirms the annotation method, the user may have some incorrect annotations, and the present application compares the phoneme sequence to determine whether the annotation is correct, avoiding artificial incorrect annotations, and therefore has higher accuracy.

[0105] Step 308: When the target phoneme sequence is not aligned with the reference phoneme sequence, determine the proportion of the uneven phonemes in the target phoneme sequence, which is the ratio of the uneven phonemes to all the phonemes in the target phoneme sequence.

[0106] Step 309: Determine the phoneme confusion vector according to the valid phoneme probability matrix.

[0107] Specifically, the confusion degree of each speech frame is calculated according to the dictionary probability vector of each speech frame, and then the confusion degrees of all speech frames are combined into a phoneme confusion degree vector.

[0108] Step 310: Select the confusion degrees of all the non-uniform phonemes from the phoneme confusion degree vector.

[0109] Step 311: When the proportion of uneven phonemes is less than or equal to a preset proportion and the confusion degrees of all uneven phonemes are less than a preset confusion degree, determine that the label of the audio data to be processed is a text sequence corresponding to the target phoneme sequence.

[0110] It can be seen from steps 308 to 311 that when the target phoneme sequence is not aligned with the reference phoneme sequence, it indicates that the two phoneme sequences are different. The preset ratio is used to determine whether the proportion of uneven phonemes is too high, and the preset ratio can be but is not limited to 0.2. When the proportion of uneven phonemes is less than or equal to the preset ratio, it indicates that the proportion of uneven phonemes is low, and there may be a small number of erroneous phonemes in the target phoneme sequence, indicating that the credibility of the target phoneme sequence is high. Then judge the confusion degree of the uneven phonemes in the target phoneme sequence. When the confusion degree of all uneven phonemes is less than the preset confusion degree, it indicates that the credibility of the text corresponding to the uneven phonemes is high. Combining the above judgment results, it can be determined that the number of uneven phonemes in the target phoneme sequence is small and the credibility of the text corresponding to the uneven phonemes is high. Therefore, it can be determined that the credibility of the target phoneme sequence is very high. The corresponding text sequence is determined according to the target phoneme sequence, and then it is used as the label of the audio data to be processed.

[0111] When the proportion of irregular phonemes is greater than a preset ratio, it indicates that the proportion of irregular phonemes is high, and the target phoneme sequence may contain many erroneous phonemes. According to the target phoneme sequence, it is determined that the audio data to be processed is noise.

[0112] When the proportion of uneven phonemes is less than or equal to the preset proportion and the confusion degrees of all uneven phonemes are greater than or equal to the preset confusion degrees, it indicates that there are a small number of erroneous phonemes in the target phoneme sequence but the characters corresponding to the uneven phonemes are easily confused. According to the target phoneme sequence, it is determined that the audio data to be processed is partially word-fuzzy audio.

[0113] In another optional embodiment, determining the phoneme confusion vector according to the valid phoneme probability matrix includes:

[0114] Step A: determining the confusion degree of the speech frame according to the dictionary probability vector of the speech frame in the valid phoneme probability matrix;

[0115] Step B: According to the order of the dictionary probability vectors in the effective phoneme probability matrix, the confusion degree of each speech frame is composed into a phoneme confusion degree vector.

[0116] In this embodiment, the effective phoneme probability matrix can be recorded as pw[B, V], and the dictionary probability vector of the b-th speech frame is recorded as pw[b, V], where b is a variable and b∈[1, B]. The confusion degree of the b-th speech frame is recorded as pm[b]. The phoneme confusion degree vector is composed according to the order of the dictionary probability vectors, and the phoneme confusion degree vector includes pm[1], pm[b2], ..., pm[b]. Among them, the order of the dictionary probability vectors in the effective phoneme probability matrix is ​​consistent with the order of the speech frames.

[0117] Optionally, step A includes:

[0118] Step A1: Arrange all dictionary probabilities in the dictionary probability vector of the speech frame in descending order.

[0119] Step A2: For the first k dictionary probabilities in the descending dictionary probability vector, a set of easily confused dictionary probabilities is selected from the first k dictionary probabilities.

[0120] The confusable dictionary probability set includes all dictionary probabilities greater than a preset probability, and k is a positive integer.

[0121] Step A3: Determine the confusion ratio according to the total number of dictionary probabilities included in the easily confused dictionary probability set and k.

[0122] Step A4: Determine the maximum probability gradient according to the maximum dictionary probability and the minimum dictionary probability among the first k dictionary probabilities.

[0123] Step A5: Determine the confusion degree of the speech frame according to the confusion ratio and the maximum probability gradient.

[0124] In this embodiment, taking the b-th speech frame as an example, the dictionary probability vector of the b-th speech frame is recorded as pw[b, V]. The dictionary probability vector of the b-th speech frame includes V dictionary probabilities, the V dictionary probabilities are arranged in descending order, and then the first k dictionary probabilities are selected from them. k is a positive integer, and its value can be set according to actual conditions.

[0125] The selected k dictionary probabilities are recorded as pw[b, k], and the dictionary probability greater than PT is selected as the easily confused dictionary probability. All easily confused dictionary probabilities can form an easily confused dictionary probability set. The number of dictionary probabilities included in the easily confused dictionary probability set is recorded as ks, and the confusion ratio r=ks / k. PT is a preset probability, and its value can be but is not limited to 3.

[0126] The maximum and minimum values ​​are selected from the set of easily confused dictionary probabilities, and the difference between the two is the maximum probability gradient Dpw. The formula for the maximum probability gradient Dpw, the maximum dictionary probability max(pw[b, k]) and the minimum dictionary probability min(pw[b, k]) can be written as: Dpw = max(pw[b, k]) - min(pw[b, k]).

[0127] The confusion degree of the bth speech frame is then determined based on the confusion ratio and the maximum probability gradient. The confusion degree pm[b], the confusion ratio r and the maximum probability gradient Dpw of the bth speech frame satisfy the following formula: pm[b]=r / Dpw.

[0128] It should be understood that the k selected dictionary probabilities are the largest k dictionary probabilities. In addition to selecting according to the above method, other equivalent methods can also be used for selection, such as arranging all dictionary probabilities in ascending order and selecting the last k dictionary probabilities. All easily confused dictionary probabilities can also form an easily confused dictionary probability sequence.

[0129] The above describes the method for determining audio data labels. The following is an introduction to the audio classification method of this application:

[0130] In another optional embodiment, when the target phoneme sequence is aligned with the reference phoneme sequence, the type of the audio data to be processed is determined to be standard audio.

[0131] In another optional embodiment, when the target phoneme sequence is not aligned with the reference phoneme sequence, the proportion of uneven phonemes in the target phoneme sequence is determined; when the proportion of uneven phonemes in the target phoneme sequence is greater than a preset ratio, the type of the audio data to be processed is determined to be noise.

[0132] In another optional embodiment, when the proportion of uneven phonemes in the target phoneme sequence is less than or equal to a preset proportion, a phoneme confusion vector is determined based on a valid phoneme probability matrix; the confusion degrees of all uneven phonemes are selected from the phoneme confusion vector; when the confusion degrees of all uneven phonemes are less than the preset confusion degree, the type of the audio data to be processed is determined to be standard audio; when the confusion degree of at least one uneven phoneme is greater than or equal to the preset confusion degree, the type of the audio data to be processed is determined to be partial word blurred audio.

[0133] In another optional embodiment, a first valid phoneme position sequence is obtained according to a valid phoneme probability matrix; when the first valid phoneme position sequence contains at least one valid phoneme position, the first valid phoneme position sequence is shifted; an average value is calculated based on the difference between a second valid phoneme position sequence obtained by the shift and the first valid phoneme position sequence; when the average value is greater than a preset threshold value, it is determined that the type of the audio signal to be processed is fast-speech audio.

[0134] In this embodiment, when the first valid grapheme position sequence includes at least one valid grapheme position, it indicates that the audio data to be processed includes one or more valid grapheme. The first valid grapheme position sequence is moved backward by one frame to obtain a second valid grapheme position sequence, and a difference sequence is obtained by subtracting the second valid grapheme position sequence from the first valid grapheme position sequence. The average value of all differences in the difference sequence is calculated, and the average value is compared with a preset threshold value. If the average value is greater than the preset threshold value, it is determined that the type of the audio signal to be processed is fast-speech audio. If the average value is less than or equal to the preset threshold value, it is determined that the type of the audio signal to be processed is not fast-speech audio.

[0135] It should be noted that, in addition to the backward movement, the first valid grapheme position sequence can also be moved forward to obtain the second valid grapheme position sequence. The number of frames moved is not limited to 1 frame, but can also be 2 frames or 3 frames, etc. The number of frames moved can be set according to the actual situation, and this application does not limit it.

[0136] In another optional embodiment, when the first valid grapheme position sequence does not include a valid grapheme position, the audio data to be processed is input into a speech activation detection model, and the speech activation detection model outputs a voicing interval according to the audio data to be processed; when the number of voicing intervals is greater than a preset number of intervals, the type of the audio data to be processed is determined to be noise; when the number of voicing intervals is less than or equal to the preset number of intervals, the audio data to be processed is determined to be invalid audio data, and the audio data to be processed is discarded. The preset number of intervals can be, but is not limited to, 2.

[0137] In the above implementation, the first automatic speech recognition model, the second automatic speech recognition model and the speech activation detection model are all deep neural network models.

[0138] The current deep learning models are all trained on standard audio data, so the recognition accuracy of the speech recognition model obtained by such training is still far from the recognition accuracy required in practice. After research, the applicant found that one of the reasons for the above gap is that the user's actual speaking speed is faster than the standard audio speed. In this regard, the present application can add fast speech audio data as training data on the basis of standard audio data to improve the recognition accuracy.

[0139] The following is an introduction to the speech recognition system. Figure 4 FIG. 2 is another schematic diagram of the speech recognition system 200 of the present application. Figure 4 In the illustrated speech recognition system 200 , the calculation module 211 may determine the type of audio data to be processed, and then store the correspondence between the audio data and the type in the data storage system 250 .

[0140] After recording a certain amount of fast-speech audio data, the training device 220 can obtain standard audio data from the database 230 and fast-speech audio data from the data storage system 250, and then use the neural network algorithm to train, verify and test the standard audio data and the fast-speech audio data, thereby updating the first automatic speech recognition model 201 or the second automatic speech recognition model 202. By learning a large amount of fast-speech audio data, the recognition accuracy can reach more than 94.6%, which can enhance the recognition ability of the automatic speech recognition model for fast-speech audio, thereby improving the recognition accuracy.

[0141] It should be understood that the training device 220 can obtain standard audio data from the database 230, obtain one or more of fast speech audio data, noise data or partially ambiguous audio data from the data storage system 250, and then use the neural network algorithm to train, verify and test the standard audio data and the audio data obtained from the data storage system 250, so as to update the first automatic speech recognition model 201 or the second automatic speech recognition model 202. In this way, the generalization ability of the automatic speech recognition model can be improved, and the automatic speech recognition model obtained by training has a higher recognition accuracy rate.

[0142] The above is an introduction to the audio data processing method of the present application. The following is an introduction to the audio processing device of the present application.

[0143] See also Figure 5 , the present application provides an audio processing device 500 including:

[0144] A first acquisition unit 501 is used to acquire audio data to be processed, where the audio data to be processed includes a plurality of speech frames;

[0145] A first speech recognition unit 502, configured to input the audio data to be processed into a first automatic speech recognition model, the first automatic speech recognition model being configured to output a phoneme probability matrix according to the audio data to be processed, the phoneme probability matrix comprising dictionary probability vectors of a plurality of speech frames;

[0146] a removing unit 503, configured to remove the dictionary probability vector of the blank frame in the phoneme probability matrix when the phoneme probability matrix includes the dictionary probability vector of the non-blank frame and the dictionary probability vector of the blank frame, so as to obtain a valid phoneme probability matrix;

[0147] A second acquisition unit 504 is used to acquire a target phoneme sequence according to a valid phoneme probability matrix;

[0148] A second speech recognition unit 505, used to input the audio data to be processed into a second automatic speech recognition model, the first automatic speech recognition model is different from the second automatic speech recognition model;

[0149] A third acquisition unit 506 is used to determine a reference phoneme sequence according to a reference text sequence output by the second automatic speech recognition model;

[0150] The label unit 507 is used to determine that the label of the audio data to be processed is the reference text sequence when the target phoneme sequence is aligned with the reference phoneme sequence.

[0151] In this embodiment, the audio processing device 500 can achieve Figure 3 The audio data processing method in the embodiment or optional embodiment. The first acquisition unit 501, the removal unit 503, the second acquisition unit 504, the third acquisition unit 506 and the label unit 507 can be implemented by a central processing unit, and the first speech recognition unit 502 and the second speech recognition unit 505 can be implemented by an artificial intelligence processor. The information interaction, execution process and other contents between the units in the audio processing device 500 are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the aforementioned method embodiment of the present application, which will not be repeated here.

[0152] In an optional embodiment, the audio processing device 500 further includes:

[0153] The first determination unit is further used to determine the proportion of irregular phonemes in the target phoneme sequence;

[0154] The second determining unit is used to determine the phoneme confusion degree vector according to the valid phoneme probability matrix; and select the confusion degrees of all the uneven phonemes from the phoneme confusion degree vector;

[0155] The label unit 507 is further configured to determine that the label of the audio data to be processed is a text sequence corresponding to the target phoneme sequence when the proportion of the uneven phonemes is less than or equal to a preset proportion and the confusion degree of all the uneven phonemes is less than a preset confusion degree. The target phoneme sequence is not aligned with the reference phoneme sequence.

[0156] In another optional embodiment, the second determination unit includes a determination subunit and a composition subunit; the determination subunit is used to determine the confusion degree of the speech frame according to the dictionary probability vector of the speech frame in the valid phoneme probability matrix; the composition subunit is used to compose the confusion degree of each speech frame into a phoneme confusion degree vector according to the order of the dictionary probability vector in the valid phoneme probability matrix.

[0157] In another optional embodiment, the determination subunit is specifically used to arrange all dictionary probabilities in the dictionary probability vector of the speech frame in descending order; for the first k dictionary probabilities in the dictionary probability vector after the descending order, select a confusing dictionary probability set from the first k dictionary probabilities, the confusing dictionary probability set includes all dictionary probabilities greater than a preset probability, k is a positive integer; determine the confusion ratio based on the total number of dictionary probabilities included in the confusing dictionary probability set and k; determine the maximum probability gradient based on the maximum dictionary probability and the minimum dictionary probability among the first k dictionary probabilities; determine the confusion degree of the speech frame based on the confusion ratio and the maximum probability gradient.

[0158] In another optional embodiment, the audio processing device 500 further includes a first classification unit;

[0159] The first classification unit is used to determine that the type of the audio data to be processed is standard audio when the target phoneme sequence is aligned with the reference phoneme sequence.

[0160] In another optional embodiment, the audio processing device 500 further includes a second classification unit;

[0161] The second classification unit is used to determine the proportion of irregular phonemes in the target phoneme sequence; when the proportion of irregular phonemes in the target phoneme sequence is greater than a preset proportion, the type of the audio data to be processed is determined to be noise. The target phoneme sequence is not aligned with the reference phoneme sequence.

[0162] In another optional embodiment, the audio processing device 500 further includes a third classification unit;

[0163] The third classification unit is used to determine the phoneme confusion vector according to the valid phoneme probability matrix; select the confusion of all the uneven phonemes from the phoneme confusion vector; when the confusion of all the uneven phonemes is less than the preset confusion, determine the type of the audio data to be processed as standard audio; when the confusion of at least one uneven phoneme is greater than or equal to the preset confusion, determine the type of the audio data to be processed as partial word fuzzy audio. The proportion of uneven phonemes in the target phoneme sequence is less than or equal to the preset proportion.

[0164] In another optional embodiment, the audio processing device 500 further includes a fourth classification unit:

[0165] The fourth classification unit is used to obtain a first valid phoneme position sequence according to a valid phoneme probability matrix; when the first valid phoneme position sequence contains at least one valid phoneme position, the first valid phoneme position sequence is shifted; an average value is calculated based on the difference between the second valid phoneme position sequence obtained by the shift and the first valid phoneme position sequence; when the average value is greater than a preset threshold, it is determined that the type of the audio signal to be processed is fast-speech audio.

[0166] In another optional embodiment, the audio processing device 500 further includes a fifth classification unit;

[0167] The fifth classification unit is used to input the audio data to be processed into the voice activation detection model, and the voice activation detection model is used to output the voicing interval according to the audio data to be processed; when the number of voicing intervals is greater than the preset number of intervals, the type of the audio data to be processed is determined to be noise; when the number of voicing intervals is less than or equal to the preset number of intervals, the audio data to be processed is discarded. The first valid grapheme phoneme position sequence does not include a valid grapheme phoneme position.

[0168] In another optional embodiment, the first automatic speech recognition model, the second automatic speech recognition model and the speech activation detection model are all deep neural network models.

[0169] In another optional embodiment, the first acquiring unit 501 is further configured to determine that the audio data to be processed is invalid audio data when the number of sampling points of the audio data to be processed does not belong to a preset interval.

[0170] In the above embodiments, the functions of the first classification unit, the second classification unit, the third classification unit, the fourth classification unit and the fifth classification unit can all be realized by the central processing unit.

[0171] The following is an introduction to the audio processing device from the perspective of hardware structure. The audio processing device takes a server as an example. Figure 6 The present application provides a server 600 including: a processor 601, a memory 602, an artificial intelligence processor 603, and a network interface 604 connected via a bus 605. The number of the processor 601, the memory 602, the artificial intelligence processor 603, and the network interface 604 can be one or more.

[0172] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0173] The memory 602 is used to store programs and data. The memory 602 may include a dynamic random access memory (DRAM) or a cache memory. For a processor such as a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., the cache memory may be integrated into the above processor.

[0174] The artificial intelligence processor 603 may be a graphics processing unit (GPU), a neural-network processing unit (NPU), an FPGA or an ASIC, etc. The artificial intelligence processor 603 may implement functions such as matrix multiplication, matrix addition, vector calculation, multiplication or division, etc.

[0175] The network interface 604 is used to receive data or send data. The network interface 604 may be a wired network interface or a wireless network interface.

[0176] The processor 601 calls the program stored in the memory 602 to implement Figure 3 The audio data processing method of the illustrated embodiment or an optional embodiment.

[0177] It should be understood that the server may also include one or more power supplies, one or more input and output devices, etc.

[0178] The present application provides a computer storage medium, including instructions, which, when executed on a computer, enables the computer to execute the audio data processing method in any one of the above embodiments or optional embodiments.

[0179] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0180] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0181] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio data processing method, characterized in that: include: Acquire audio data to be processed, wherein the audio data to be processed includes a plurality of speech frames; Inputting the audio data to be processed into a first automatic speech recognition model, wherein the first automatic speech recognition model is used to output a phoneme probability matrix according to the audio data to be processed, wherein the phoneme probability matrix includes dictionary probability vectors of the multiple speech frames; When the phoneme probability matrix includes dictionary probability vectors of non-blank frames and dictionary probability vectors of blank frames, removing the dictionary probability vectors of blank frames in the phoneme probability matrix to obtain a valid phoneme probability matrix; Acquire a target phoneme sequence according to the valid phoneme probability matrix; Inputting the to-be-processed audio data into a second automatic speech recognition model, wherein the first automatic speech recognition model is different from the second automatic speech recognition model; Determine a reference phoneme sequence according to a reference text sequence output by the second automatic speech recognition model; When the target phoneme sequence is aligned with the reference phoneme sequence, it is determined that the label of the audio data to be processed is the reference text sequence.

2. The method according to claim 1, characterized in that: The target phoneme sequence is not aligned with the reference phoneme sequence; The method further comprises: Determining the proportion of irregular phonemes in the target phoneme sequence; Determine a phoneme confusion vector according to the valid phoneme probability matrix; Selecting the confusion degrees of all the irregular phonemes from the phoneme confusion degree vector; When the proportion of the irregular phonemes is less than or equal to a preset proportion and the confusion degrees of all the irregular phonemes are less than a preset confusion degree, it is determined that the label of the audio data to be processed is a text sequence corresponding to the target phoneme sequence.

3. The method according to claim 2, characterized in that Determining the phoneme confusion vector according to the valid phoneme probability matrix comprises: Determining the confusion degree of the speech frame according to the dictionary probability vector of the speech frame in the valid phoneme probability matrix; According to the order of the dictionary probability vectors in the effective phoneme probability matrix, the confusion degree of each speech frame is combined into a phoneme confusion degree vector.

4. The method according to claim 3, characterized in that Determining the confusion degree of the speech frame according to the dictionary probability vector of the speech frame in the valid phoneme probability matrix comprises: Arrange all dictionary probabilities in the dictionary probability vector of the speech frame in descending order; For the first k dictionary probabilities in the dictionary probability vector after descending order, select a confusing dictionary probability set from the first k dictionary probabilities, the confusing dictionary probability set includes all dictionary probabilities greater than a preset probability, and k is a positive integer; Determine the confusion ratio according to the total number of dictionary probabilities included in the easily confusing dictionary probability set and k; Determine a maximum probability gradient according to the maximum dictionary probability and the minimum dictionary probability among the first k dictionary probabilities; The confusion degree of the speech frame is determined according to the confusion ratio and the maximum probability gradient.

5. The method according to claim 1, characterized in that The method further comprises: When the target phoneme sequence is aligned with the reference phoneme sequence, it is determined that the type of the audio data to be processed is standard audio.

6. The method according to claim 1, characterized in that The target phoneme sequence is not aligned with the reference phoneme sequence; Determining the proportion of irregular phonemes in the target phoneme sequence; When the proportion of irregular phonemes in the target phoneme sequence is greater than a preset proportion, it is determined that the type of the audio data to be processed is noise.

7. The method according to claim 6, characterized in that The proportion of uneven phonemes in the target phoneme sequence is less than or equal to a preset proportion; The method further comprises: Determine a phoneme confusion vector according to the valid phoneme probability matrix; Selecting the confusion degrees of all the irregular phonemes from the phoneme confusion degree vector; When the confusion degree of all the irregular phonemes is less than a preset confusion degree, determining that the type of the audio data to be processed is standard audio; When the confusion degree of at least one irregular phoneme is greater than or equal to a preset confusion degree, it is determined that the type of the audio data to be processed is partial word ambiguous audio.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Acquire a first valid word phoneme position sequence according to the valid phoneme probability matrix; When the first valid grapheme phoneme position sequence includes at least one valid grapheme phoneme position, shifting the first valid grapheme phoneme position sequence; Calculating an average value based on the difference between the second effective grapheme position sequence obtained by shifting and the first effective grapheme position sequence; When the average value is greater than a preset threshold, it is determined that the type of the audio data to be processed is fast-speech audio.

9. The method according to claim 8, characterized in that The first valid grapheme-phoneme position sequence does not include a valid grapheme-phoneme position; The method further comprises: Inputting the audio data to be processed into a speech activation detection model, wherein the speech activation detection model is used to output a voicing interval according to the audio data to be processed; When the number of the sounding intervals is greater than the preset number of intervals, determining that the type of the audio data to be processed is noise; When the number of utterance intervals is less than or equal to the preset number of intervals, the audio data to be processed is discarded.

10. The method according to claim 9, characterized in that The first automatic speech recognition model, the second automatic speech recognition model and the speech activation detection model are all deep neural network models.

11. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When the number of sampling points of the audio data to be processed does not belong to a preset interval, it is determined that the audio data to be processed is invalid audio data.

12. An audio processing device, characterized in that: include: A first acquisition unit, configured to acquire audio data to be processed, wherein the audio data to be processed includes a plurality of speech frames; A first speech recognition unit, configured to input the audio data to be processed into a first automatic speech recognition model, wherein the first automatic speech recognition model is configured to output a phoneme probability matrix according to the audio data to be processed, wherein the phoneme probability matrix includes dictionary probability vectors of the plurality of speech frames; a removing unit, configured to remove the dictionary probability vector of the blank frame in the phoneme probability matrix when the phoneme probability matrix includes the dictionary probability vector of the non-blank frame and the dictionary probability vector of the blank frame, so as to obtain a valid phoneme probability matrix; A second acquisition unit, configured to acquire a target phoneme sequence according to the valid phoneme probability matrix; A second speech recognition unit, configured to input the to-be-processed audio data into a second automatic speech recognition model, wherein the first automatic speech recognition model is different from the second automatic speech recognition model; A third acquisition unit, configured to determine a reference phoneme sequence according to a reference text sequence output by the second automatic speech recognition model; A labeling unit is used to determine that the label of the audio data to be processed is the reference text sequence when the target phoneme sequence is aligned with the reference phoneme sequence.

13. The device according to claim 12, characterized in that The target phoneme sequence is not aligned with the reference phoneme sequence; The audio processing device further comprises: The first determining unit is further used to determine the proportion of irregular phonemes in the target phoneme sequence; A second determining unit is used to determine a phoneme confusion degree vector according to the valid phoneme probability matrix; and select confusion degrees of all the irregular phonemes from the phoneme confusion degree vector; The labeling unit is also used to determine that the label of the audio data to be processed is the text sequence corresponding to the target phoneme sequence when the proportion of the uneven phonemes is less than or equal to a preset proportion and the confusion degrees of all the uneven phonemes are less than a preset confusion degree.

14. The device according to claim 13, characterized in that The second determining unit includes: A determination subunit, configured to determine the confusion degree of a speech frame according to a dictionary probability vector of the speech frame in the valid phoneme probability matrix; The composition subunit is used to compose the confusion degree of each speech frame into a phoneme confusion degree vector according to the order of the dictionary probability vectors in the effective phoneme probability matrix.

15. The device according to claim 14, characterized in that The determination subunit is specifically used to arrange all dictionary probabilities in the dictionary probability vector of the speech frame in descending order; for the first k dictionary probabilities in the dictionary probability vector after descending order, select a confusing dictionary probability set from the first k dictionary probabilities, the confusing dictionary probability set includes all dictionary probabilities greater than a preset probability, and k is a positive integer; determine the confusion ratio according to the total number of dictionary probabilities included in the confusing dictionary probability set and k; determine the maximum probability gradient according to the maximum dictionary probability and the minimum dictionary probability among the first k dictionary probabilities; and determine the confusion degree of the speech frame according to the confusion ratio and the maximum probability gradient.

16. The device according to claim 12, characterized in that The audio processing device further comprises: The first classification unit is used to determine that the type of the audio data to be processed is standard audio when the target phoneme sequence is aligned with the reference phoneme sequence.

17. The device according to claim 12, characterized in that The target phoneme sequence is not aligned with the reference phoneme sequence; The audio processing device further comprises: The second classification unit is used to determine the proportion of uneven phonemes in the target phoneme sequence; when the proportion of uneven phonemes in the target phoneme sequence is greater than a preset proportion, determine that the type of the audio data to be processed is noise.

18. The device according to claim 17, characterized in that The proportion of uneven phonemes in the target phoneme sequence is less than or equal to a preset proportion; The audio processing device further comprises: A third classification unit is used to determine a phoneme confusion degree vector according to the valid phoneme probability matrix; select the confusion degrees of all irregular phonemes from the phoneme confusion degree vector; when the confusion degrees of all irregular phonemes are less than a preset confusion degree, determine that the type of the audio data to be processed is standard audio; When the confusion degree of at least one irregular phoneme is greater than or equal to a preset confusion degree, it is determined that the type of the audio data to be processed is partial word ambiguous audio.

19. The device according to any one of claims 12 to 18, characterized in that The audio processing device further comprises: The fourth classification unit is used to obtain a first valid phoneme position sequence according to the valid phoneme probability matrix; when the first valid phoneme position sequence contains at least one valid phoneme position, the first valid phoneme position sequence is shifted; an average value is calculated based on the difference between the second valid phoneme position sequence obtained by the shift and the first valid phoneme position sequence; when the average value is greater than a preset threshold, it is determined that the type of the audio data to be processed is fast-speech audio.

20. The device according to claim 19, characterized in that The first valid grapheme-phoneme position sequence does not include a valid grapheme-phoneme position; The audio processing device further comprises: A fifth classification unit is used to input the audio data to be processed into a speech activation detection model, and the speech activation detection model is used to output the vocalization interval according to the audio data to be processed; when the number of the vocalization intervals is greater than the preset number of intervals, determine that the type of the audio data to be processed is noise; when the number of the vocalization intervals is less than or equal to the preset number of intervals, discard the audio data to be processed.

21. The device according to claim 20, characterized in that The first automatic speech recognition model, the second automatic speech recognition model and the speech activation detection model are all deep neural network models.

22. The device according to any one of claims 12 to 18, characterized in that The first acquisition unit is further configured to determine that the audio data to be processed is invalid audio data when the number of sampling points of the audio data to be processed does not belong to a preset interval.

23. A computer storage medium comprising instructions, characterized in that: When the method is executed on a computer, the computer is enabled to execute the audio data processing method according to any one of claims 1 to 11.

Images