Audio category prediction method, device, storage medium and electronic device

Abstract

The present application discloses an audio category prediction method, device, storage medium and electronic device, which relates to the field of artificial intelligence technology. The method includes: obtaining acoustic feature information and phoneme alignment information of at least one audio, and calibrating the corresponding audio category of each audio; using an audio analysis model to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio to obtain the deep pronunciation features of each audio; compressing and mapping the deep pronunciation features of audios of the same audio category to obtain at least one compressed feature set; constructing a Gaussian process based on at least one compressed feature set to obtain an audio category prediction function; adjusting the parameters in the audio analysis model based on the audio category prediction function to obtain the trained audio analysis model to predict the audio category of the audio to be analyzed. The present application effectively reduces the computational complexity in the audio category prediction work while improving the audio category prediction effect.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to an audio category prediction method, device, storage medium and electronic device. Background Art

[0002] Audio category prediction is the work of analyzing the category of audio, such as the work of scoring audio in an oral exam. Due to various factors, such as the pronunciation of different speaking regions, audio recording quality, audio recording environment, etc., audio category prediction is very difficult.

[0003] At present, in the related technical solutions, a large number of audio samples are collected for large-scale learning, which has the problem of high computational complexity, or some audio samples are selected from a large number of audio samples for learning, and the extracted samples are unreliable, resulting in poor audio category prediction effect.

[0004] Therefore, there are currently problems in which the audio category prediction work has high computational complexity and the audio category prediction effect is poor. Summary of the invention

[0005] The embodiments of the present application provide an audio category prediction method and related devices, which can effectively reduce the computational complexity in the audio category prediction work while improving the audio category prediction effect.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] According to one embodiment of the present application, a method for predicting an audio category includes: obtaining acoustic feature information and phoneme alignment information of at least one audio, and calibrating the corresponding audio category of each audio; using an audio analysis model to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio to obtain deep pronunciation features of each audio; performing compression mapping processing on the deep pronunciation features of the audio of the same audio category to obtain at least one compressed feature set; constructing a Gaussian process based on the at least one compressed feature set to obtain an audio category prediction function; adjusting the parameters in the audio analysis model based on the audio category prediction function to obtain the trained audio analysis model to predict the audio category of the audio to be analyzed.

[0008] According to one embodiment of the present application, an audio category prediction device includes: an acquisition module, which is used to acquire acoustic feature information and phoneme alignment information of at least one audio, and each audio is calibrated to a corresponding audio category; an input module, which is used to use an audio analysis model to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio, so as to obtain deep pronunciation features of each audio; a compression module, which is used to compress and map the deep pronunciation features of the audio of the same audio category, so as to obtain at least one compressed feature set; a Gaussian module, which is used to construct a Gaussian process based on the at least one compressed feature set to obtain an audio category prediction function; a prediction module, which is used to adjust the parameters in the audio analysis model based on the audio category prediction function to obtain the trained audio analysis model to predict the audio category of the audio to be analyzed.

[0009] In some embodiments of the present application, the acoustic feature information of the audio includes at least one frame of sub-acoustic features of the audio, and the phoneme alignment information of the audio includes the pronunciation start and end time periods of the words in the audio; the input module includes: a depth extraction unit, which is used to perform deep extraction processing on at least one frame of sub-acoustic features of each of the audios to obtain at least one frame of deep features corresponding to each of the audios; a first normalization unit, which is used to normalize the depth features of at least one frame of deep features corresponding to each of the audios that belong to the same pronunciation start and end time period to obtain the word features of each word in the audio; a second normalization unit, which is used to normalize the word features of each word in the audio to obtain the deep pronunciation features of each of the audios.

[0010] In some embodiments of the present application, the deep extraction unit includes: a network input subunit, used to input at least one frame of sub-acoustic features of each of the audio into a feature extraction neural network; a network extraction subunit, used to perform deep extraction processing on the input at least one frame of sub-acoustic features based on the feature extraction neural network to obtain at least one frame of deep features corresponding to each of the audio.

[0011] In some embodiments of the present application, the compression module includes: a clustering unit, which is used to cluster the deep pronunciation features of each audio according to the audio category corresponding to each audio, so as to obtain at least one feature cluster, each of which includes the deep pronunciation features of at least one audio, and each of which corresponds to an audio category; a mapping unit, which is used to compress and map each of the feature clusters into a corresponding compressed feature set, so as to obtain at least one compressed feature set, and the feature dimension of each of the compressed feature sets is smaller than that of the corresponding feature cluster.

[0012] In some embodiments of the present application, the mapping unit includes: a mapping input subunit, used to input each of the feature clusters into a compression mapping neural network respectively; and a network implicit unit, used to compress and map the input feature cluster into a corresponding compression feature set based on the compression mapping neural network.

[0013] In some embodiments of the present application, the Gaussian module includes: a sample construction module, which is used to construct a training feature set and a test feature set based on the compressed features in the at least one compressed feature set; a matrix generation unit, which is used to perform covariance operation processing on the compressed features in the training feature set and the test feature set based on the covariance function to generate a target covariance matrix; a mean generation unit, which is used to perform mean operation processing on the compressed features in the training feature set and the test feature set based on the mean function to generate a target mean vector; and a function construction unit, which is used to construct the audio category prediction function based on the covariance matrix and the target mean vector.

[0014] In some embodiments of the present application, the matrix generation unit includes: a first covariance generation subunit, used to perform covariance operation processing on the compressed features in the training feature set based on the covariance function to obtain a first covariance matrix; a second covariance generation subunit, used to perform covariance operation processing on the compressed features in the test feature set based on the covariance function to obtain a second covariance matrix; a third generation subunit, used to perform covariance operation processing on the compressed features in the training feature set and the test feature set based on the covariance function to obtain a third covariance covariance matrix; a matrix determination subunit, used to use the first covariance matrix, the second covariance matrix and the third covariance matrix as the target covariance matrix.

[0015] In some embodiments of the present application, the mean generation unit includes: a first mean generation subunit, used to perform mean operation processing on the compressed features in the training feature set based on the mean function to obtain a first mean vector; a second mean generation subunit, used to perform mean operation processing on the compressed features in the training feature set based on the mean function to obtain a second mean vector; and a mean determination subunit, used to use the first mean vector and the second mean vector as the target mean vector.

[0016] In some embodiments of the present application, the function construction unit includes: a distribution function acquisition subunit, used to obtain an audio category distribution function that satisfies a Gaussian distribution and is generated based on the mean function and the covariance function; a training category acquisition subunit, used to obtain the audio category corresponding to the compressed feature in the training feature set; and a prediction function construction subunit, used to construct an audio category prediction function that predicts the audio category corresponding to the compressed feature in the test feature set based on the audio category distribution function, the audio category corresponding to the compressed feature in the training feature set, the target covariance matrix and the target mean vector.

[0017] In some embodiments of the present application, the prediction module includes: a target posterior probability determination unit, used to determine the target posterior probability of the audio category predicted by the audio category prediction function; and an estimation unit, used to perform maximum likelihood estimation based on the target posterior probability to adjust the parameters in the audio analysis model.

[0018] In some embodiments of the present application, the target posterior probability determination unit includes: a first probability acquisition subunit, used to obtain the prior probability of the audio category predicted by the audio category prediction function; a second probability acquisition subunit, used to determine the posterior probability of the audio category predicted by the audio category prediction function; and a third probability acquisition subunit, used to take the product of the prior probability and the posterior probability as the target posterior probability.

[0019] In some embodiments of the present application, the audio to be analyzed includes spoken audio that follows a target text; the prediction module includes: a spoken information extraction unit, used to extract acoustic feature information and phoneme alignment information corresponding to the spoken audio based on the target text; a spoken classification unit, used to input the acoustic feature information and phoneme alignment information corresponding to the spoken audio into the trained audio analysis model to output the audio category of the spoken audio.

[0020] According to another embodiment of the present application, a storage medium stores computer-readable instructions thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer executes the method described in the embodiment of the present application.

[0021] According to another embodiment of the present application, an electronic device includes: a memory storing computer-readable instructions; and a processor reading the computer-readable instructions stored in the memory to execute the method described in the embodiment of the present application.

[0022] According to another embodiment of the present application, a computer program product or a computer program includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0023] In an embodiment of the present application, acoustic feature information and phoneme alignment information of at least one audio are obtained, and the audio category corresponding to each audio is calibrated; an audio analysis model is used to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio to obtain deep pronunciation features of each audio; the deep pronunciation features of audios of the same audio category are compressed and mapped to obtain at least one compressed feature set; a Gaussian process is constructed based on at least one compressed feature set to obtain an audio category prediction function; and parameters in the audio analysis model are adjusted based on the audio category prediction function to obtain the trained audio analysis model to predict the audio category of the audio to be analyzed.

[0024] In this way, at least one compressed feature set is obtained based on deep feature extraction processing and compression mapping processing as the input of the Gaussian process to construct the Gaussian process, and a sparse Gaussian deep kernel learning process is established, which can effectively reduce the computational complexity while ensuring the accuracy of audio category prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A schematic diagram of a system to which the embodiments of the present application can be applied is shown.

[0027] Figure 2 A schematic diagram showing another system to which the embodiments of the present application can be applied is shown.

[0028] Figure 3 A flowchart of an audio category prediction method according to an embodiment of the present application is shown.

[0029] Figure 4 A flow chart of a deep feature extraction method according to an embodiment of the present application is shown.

[0030] Figure 5 A schematic diagram of a deep feature extraction network according to an embodiment of the present application is shown.

[0031] Figure 6 A flowchart of compression mapping processing according to an embodiment of the present application is shown.

[0032] Figure 7 A flowchart of performing likelihood estimation according to an embodiment of the present application is shown.

[0033] Figure 8 A terminal interface diagram showing an audio category prediction process in a scenario is shown.

[0034] Fig. 9 Another terminal interface diagram showing an audio category prediction process in a scenario.

[0035] Fig.10 Another terminal interface diagram showing an audio category prediction process in a scenario.

[0036] Fig.11 A block diagram of an audio category prediction device according to another embodiment of the present application is shown.

[0037] Fig.12 A block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0039] In the following description, the specific embodiments of the present application will be described with reference to the steps and symbols performed by one or more computers, unless otherwise stated. Therefore, these steps and operations will be mentioned several times as being performed by a computer, and the computer execution referred to herein includes the operation of a computer processing unit by an electronic signal representing data in a structured form. This operation converts the data or maintains it at a location in the memory system of the computer, which can be reconfigured or otherwise change the operation of the computer in a manner familiar to testers in the field. The data structure maintained by the data is a physical location in the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the above text, which does not represent a limitation, and testers in the field will understand that the various steps and operations described below can also be implemented in hardware.

[0040] Figure 1 FIG. 1 is a schematic diagram showing a system 100 to which an embodiment of the present application can be applied. Figure 1As shown, the system 100 may include a server 101 and a terminal 102. The server 101 and the terminal 102 may be directly or indirectly connected via wireless communication, and this application does not impose any special limitation thereto.

[0041] Data can be transmitted between the server 101 and the terminal 102 via a target protocol link, which may include a link based on a transport layer protocol, such as a Transmission Control Protocol (TCP) link or a User Datagram Protocol (UDP) link transmission and other transport layer protocols.

[0042] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0043] In one implementation of this example, server 101 is a cloud server, and server 101 can provide artificial intelligence cloud services, such as artificial intelligence cloud services for massively multiplayer online role-playing games (MMORPGs). The so-called artificial intelligence cloud service is generally referred to as AIaaS (AI as a Service, "AI as a Service" in Chinese). This is a mainstream service mode of an artificial intelligence platform. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access one or more artificial intelligence services provided by the platform through an API interface. Some senior developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and operate their own cloud artificial intelligence services. For example, server 101 can provide an audio category prediction service based on artificial intelligence.

[0044] The terminal 102 can be any device, including but not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, VR / AR devices, smart watches, computers, etc.

[0045] In one implementation of this example, the server 101 can obtain acoustic feature information and phoneme alignment information of at least one audio, and calibrate the corresponding audio category of each audio; use an audio analysis model to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio to obtain deep pronunciation features of each audio; perform compression mapping processing on the deep pronunciation features of audios of the same audio category to obtain at least one compressed feature set; construct a Gaussian process based on at least one compressed feature set to obtain an audio category prediction function; adjust the parameters in the audio analysis model based on the audio category prediction function to obtain the trained audio analysis model to predict the audio category of the audio to be analyzed.

[0046] In one example, reference Figure 1 The server 101 can send the obtained relevant information of the audio to be analyzed (including the audio to be analyzed and the text corresponding to the audio to be analyzed) to the speech recognition module, obtain the acoustic feature information and phoneme alignment information of the audio to be analyzed, and then send the acoustic feature information and phoneme alignment information of the audio to be analyzed to the audio category prediction module, and use the trained audio analysis model therein to predict the audio category of the audio to be analyzed.

[0047] Figure 2 FIG. 2 is a schematic diagram of another system 200 to which the embodiments of the present application can be applied. Figure 2 As shown, the system 200 may be a distributed system formed by a client 201 and multiple nodes 202 connected via network communication.

[0048] Take the distributed system as the blockchain system as an example, see Figure 2 , Figure 2 This is an optional structural diagram of a distributed system 200 provided in an embodiment of the present application applied to a blockchain system, which is formed by multiple nodes 202 and a client 201. The nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In a distributed system, any machine such as a server can join and become a node 202 (each node 202 can be such as Figure 1 In the server 101), the node 202 can provide an audio category prediction service, and the node includes a hardware layer, an intermediate layer, an operating system layer and an application layer.

[0049] See also Figure 2 The functions of each node in the blockchain system shown include:

[0050] 1) Routing: a basic function of a node, used to support communication between nodes.

[0051] In addition to the routing function, the node can also have the following functions:

[0052] 2) Applications are deployed in the blockchain to implement specific businesses based on actual business needs, record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to add the record data to a temporary block when they successfully verify the source and integrity of the record data.

[0053] For example, the services implemented by the application include:

[0054] 2.1) Wallet, used to provide the function of conducting electronic currency transactions, including initiating transactions (i.e., sending the transaction record of the current transaction to other nodes in the blockchain system. After the other nodes successfully verify the transaction, as a response to acknowledge the validity of the transaction, the transaction record data is stored in the temporary block of the blockchain; of course, the wallet also supports querying the remaining electronic currency in the electronic currency address;

[0055] 2.2) Shared ledger, which is used to provide functions such as storage, query and modification of account data. The record data of the operation on the account data is sent to other nodes in the blockchain system. After other nodes verify the validity, as a response to acknowledging the validity of the account data, the record data is stored in a temporary block, and a confirmation can also be sent to the node that initiated the operation.

[0056] 2.3) Smart contracts are computerized protocols that can execute the terms of a contract. They are implemented by deploying code on a shared ledger that is executed when certain conditions are met. The code is used to complete automated transactions based on actual business needs, such as querying the logistics status of the goods purchased by the buyer and transferring the buyer's electronic currency to the merchant's address after the buyer signs for the goods. Of course, smart contracts are not limited to executing contracts for transactions, but can also execute contracts for processing received information.

[0057] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0058] In one implementation of this example, node 202 can obtain acoustic feature information and phoneme alignment information of at least one audio, and calibrate the corresponding audio category of each audio; use an audio analysis model to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio to obtain deep pronunciation features of each audio; perform compression mapping processing on the deep pronunciation features of audios of the same audio category to obtain at least one compressed feature set; construct a Gaussian process based on at least one compressed feature set to obtain an audio category prediction function; adjust the parameters in the audio analysis model based on the audio category prediction function to obtain the trained audio analysis model to predict the audio category of the audio to be analyzed.

[0059] Figure 3 The flowchart of the audio category prediction method according to an embodiment of the present application is schematically shown. The execution subject of the audio category prediction method can be any terminal, such as Figure 1 The server 101 shown or Figure 2 Node 202 is shown.

[0060] like Figure 3 As shown, the audio category prediction method may include steps S310 to S350.

[0061] Step S310, obtaining acoustic feature information and phoneme alignment information of at least one audio, and calibrating the corresponding audio category of each audio;

[0062] Step S320, using an audio analysis model to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio, to obtain deep pronunciation features of each audio;

[0063] Step S330, performing compression mapping processing on the deep pronunciation features of the audios of the same audio category to obtain at least one compressed feature set;

[0064] Step S340, constructing a Gaussian process based on at least one compressed feature set to obtain an audio category prediction function;

[0065] Step S350: adjusting the parameters in the audio analysis model based on the audio category prediction function to obtain the audio category of the audio to be analyzed predicted by the trained audio analysis model.

[0066] The following describes the specific process of each step when predicting the audio category.

[0067] In step S310, acoustic feature information and phoneme alignment information of at least one audio are obtained, and each audio is calibrated to a corresponding audio category.

[0068] In the implementation of this example, the audio is speech, such as spoken language or singing, and the audio category is a label describing the category to which the audio belongs. It can be understood that the audio category can be calibrated according to requirements. The number of audios in at least one audio can be selected according to requirements. At least one audio may include audio of at least one audio category, and each audio category may include a predetermined number of audios. For example, at least one audio includes 500, the audio categories may include five, and each audio category may include 100 audios.

[0069] In one example, the audio includes spoken audio (eg, English spoken audio) of the target text, and the audio categories may include five types: one star, two stars, three stars, four stars, and five stars. The higher the star rating, the higher the corresponding score of the audio.

[0070] Acoustic feature information is a physical quantity that represents the acoustic characteristics of audio. It is also a general term for the acoustic performance of various sound elements. Acoustic feature information includes frequency cepstrum coefficients (MFCC), fundamental frequency characteristics, and formant characteristics. Phoneme alignment information is the pronunciation time alignment information of words in the audio. Phoneme alignment information may include the pronunciation start and end time period corresponding to each word in the audio (i.e., the time period between the pronunciation start time and the pronunciation end time corresponding to each word). For example, the pronunciation start and end time period of a certain word is 1.5s to 2 seconds.

[0071] When obtaining the acoustic feature information and phoneme alignment information of each audio, the audio and the text corresponding to the audio can be collected first; then, the audio signal of the audio is subjected to acoustic feature extraction, such as extracting the frequency cepstrum coefficient (MFCC) of the audio signal, and the extracted acoustic features include a sub-acoustic feature corresponding to each frame, and the feature start and end time period corresponding to the sub-acoustic feature of each frame in the audio signal (the time period between the start time and the end time corresponding to the sub-acoustic feature) can be recorded; the audio and the text corresponding to the audio are subjected to phoneme alignment through speech recognition, and the pronunciation start and end time period corresponding to the words in the audio in the audio signal is obtained as the phoneme alignment information. Among them, the audio and the text corresponding to the audio can be sent to a trained information extractor based on speech recognition technology (Automatic Speech Recognition, ASR) for extraction to obtain the acoustic feature information and phoneme alignment information of the audio.

[0072] In step S320, an audio analysis model is used to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio to obtain the deep pronunciation features of each audio.

[0073] In the implementation of this example, the audio analysis model is a model to be trained for audio category prediction, and the audio analysis model may include a depth extraction processing unit, a compression mapping processing unit, and a Gaussian process construction unit.

[0074] Deep extraction processing is the process of extracting deep pronunciation features that represent richer audio information than acoustic feature information. The deep extraction processing unit can perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio to obtain the deep pronunciation features of each audio.

[0075] Based on the deep extraction and processing of the acoustic feature information and phoneme alignment information of each audio, the deep pronunciation features of each audio are obtained. When the pseudo sample (i.e., compressed feature set) is constructed based on the compression mapping processing in the subsequent steps, the acoustic feature information can be sparsely constructed while the pseudo sample has excellent audio information representation capabilities.

[0076] In one embodiment, the acoustic feature information of the audio includes at least one sub-frame acoustic feature of the audio, and the phoneme alignment information of the audio includes the pronunciation start and end time periods of the words in the audio; in step S320, the acoustic feature information and phoneme alignment information of each audio are subjected to deep extraction processing to obtain the deep pronunciation features of each audio, including:

[0077] Perform deep extraction processing on at least one frame of sub-acoustic features of each audio to obtain at least one frame of deep features corresponding to each audio; normalize the deep features belonging to the same pronunciation start and end time period in at least one frame of deep features corresponding to each audio to obtain word features of each word in the audio; normalize the word features of each word in the audio to obtain deep pronunciation features of each audio.

[0078] The acoustic feature information of the audio includes at least one frame of sub-acoustic features of the audio, that is, the acoustic feature information of each audio is composed of a frame of sub-acoustic features, and the acoustic feature information is at the frame level. Each frame of sub-acoustic features can include multi-element features, for example, a sub-acoustic feature can be a three-element feature of [1, 0.2, 3], where 1 is the initial pronunciation feature of the phoneme, 0.2 is the middle stable pronunciation feature of the phoneme, and 3 is the tail pronunciation feature of the phoneme.

[0079] Deep extraction processing is performed on at least one frame of sub-acoustic features of each audio. For each audio, deep extraction processing can be performed on at least one frame of sub-acoustic features included in the acoustic feature information of the audio to obtain the depth features corresponding to each frame of sub-acoustic features, and then obtain at least one frame of depth features corresponding to the audio.

[0080] Among them, when performing deep extraction processing on at least one frame of sub-acoustic features of audio, in one method, at least one frame of sub-acoustic features of audio can be deeply extracted by convolution processing to obtain at least one frame of sub-acoustic features of audio. The convolution processing method is, for example, first constructing a feature matrix of at least one frame of sub-acoustic features, and each frame of sub-acoustic features corresponds to a position in the feature matrix. Then, the feature matrix can be convolved based on a preset convolution matrix to obtain a convolution result matrix, and the feature value of the corresponding position of each frame of sub-acoustic features is determined from the convolution result matrix as the deep feature corresponding to each frame of sub-acoustic features. In another method, at least one frame of sub-acoustic features of audio can be deeply extracted by long-term and short-term memory fusion processing to obtain at least one frame of sub-acoustic features of audio. The long-term and short-term memory fusion processing method is, for example, aggregating partial features of other frame sub-acoustic features for each frame of sub-acoustic features (for example, aggregating by dot multiplication, etc.) to obtain the deep feature corresponding to each frame of sub-acoustic features. It can be understood that in other embodiments, at least one frame of sub-acoustic features of audio can be deeply extracted by fusion methods based on attention mechanisms.

[0081] For example, see Figure 4 The acoustic feature information of the audio “I like apple” includes six frames of sub-acoustic features [1, 0.2, 3], [1.2, 3, 0.5], [0.3, 2, 3], [0.2, 3, 0.4], [1, 1.2, 3] and [2, 3.5, 4]. By performing deep extraction processing on the six frames of sub-acoustic features in the acoustic feature information of the audio “I like apple”, six frames of deep features [1, 2, 3], [2, 3, 4], [1, 1, 3], [1, 3, 4], [4, 2, 3] and [2, 3, 4] can be obtained.

[0082] When extracting each frame of sub-acoustic features, the feature start and end time periods corresponding to the sub-acoustic features in the audio signal (the time period between the start time and the end time corresponding to the sub-acoustic features) can be recorded, and then the feature start and end time periods corresponding to the deep features corresponding to each frame of sub-acoustic features can be determined. Furthermore, based on the pronunciation start and end time periods of each word, the deep features belonging to the same pronunciation start and end time periods can be determined.

[0083] For example, see Figure 4In the audio 'I like apple', the pronunciation start and end time period corresponding to I can be 1-1.5 seconds. According to the feature start and end time periods corresponding to the sub-acoustic features, the sub-acoustic features [1, 0.2, 3] and [1.2, 3, 0.5] can be determined to belong to the same pronunciation start and end time period corresponding to I. Furthermore, the depth feature [1, 2, 3] corresponding to the sub-acoustic feature [1, 0.2, 3] and the depth feature [2, 3, 4] corresponding to the sub-acoustic feature [1.2, 3, 0.5] can be determined to belong to the same pronunciation start and end time period corresponding to I.

[0084] The deep features belonging to the same pronunciation start and end time period are normalized, which can be to average or sum the deep features belonging to the same pronunciation start and end time period. In this example, the normalization process chooses averaging. For example, the deep features [1, 2, 3] corresponding to I and the meta-features in [2, 3, 4] are averaged respectively to obtain the word features [1.5, 2.5, 3.5] corresponding to I.

[0085] Finally, the word features of the words in the audio can be normalized by averaging or summing them. In this example, the averaging method is chosen. For example, the word features of I are [1.5, 2.5, 3.5], the word features of like are [1, 2, 3.5], and the word features of apple are [3, 2.5, 3.5]. The average of each meta-feature is used to obtain the deep pronunciation features of the audio "I like apple" [1.53, 2.33, 3.5].

[0086] In one embodiment, performing deep extraction processing on at least one frame of sub-acoustic features of each audio to obtain at least one frame of deep features corresponding to each audio includes:

[0087] Input at least one frame of sub-acoustic features of each audio into a feature extraction neural network; perform deep extraction processing on the input at least one frame of sub-acoustic features based on the feature extraction neural network to obtain at least one frame of deep features corresponding to each audio.

[0088] The feature extraction neural network can be a convolutional neural network (which can perform convolution processing), a long short-term memory network (which can perform long short-term memory fusion processing), and other neural networks. The feature extraction neural network can be a superposition of multiple layers of neural networks, or a superposition of multiple layers of the same or different neural networks, such as a superposition of three layers of convolutional networks as a feature extraction neural network. Based on the feature extraction neural network, deep extraction processing such as convolution processing or long short-term memory fusion processing is performed. The feature extraction neural network serves as a deep extraction processing unit in the audio analysis model, which can ensure the stability of the audio analysis model for audio category prediction.

[0089] Among them, the feature extraction neural network can be a pre-trained feature extraction neural network. The pre-training process of the feature extraction neural network can include: for the target feature extraction model as shown in Figure 5 , the target feature extraction model can include a feature extraction neural network and a fully connected network; an acoustic feature information set can be obtained. Each acoustic feature information in the acoustic feature information set includes at least one frame of sub-acoustic features, and each sub-acoustic feature is labeled with a feature label (senone label); then, based on each acoustic feature information in the acoustic feature information set as the input feature in the target feature extraction model, deep features are extracted based on the feature extraction neural network, and the deep features are fully connected processed based on the fully connected network, and the feature label (senone label) of each sub-acoustic feature in the acoustic feature information is output, and the target feature extraction model is iteratively trained until the accuracy of the target feature extraction model in predicting the feature label (senone label) meets the prediction accuracy requirement, and then the feature extraction neural network in the trained target feature extraction model is set as the feature extraction neural network in the audio analysis model, which can further reduce the computational complexity of the audio category prediction work.

[0090] In step S330, the deep pronunciation features of the audio of the same audio category are subjected to compression mapping processing to obtain at least one compressed feature set.

[0091] In the implementation manner of this example, at least one audio can include at least one type of audio category. Each audio category can include a predetermined number of audios. The audios included in each audio category are the audios of the same audio category. For example, all the audios corresponding to the audio category of one star are the audios of the same audio category.

[0092] The compression mapping process is a process of compressing and mapping the deep pronunciation feature set (the set composed of the deep pronunciation features of the audio of the same audio category) into a compressed feature set with a lower feature dimension. In the compression mapping processing unit in the audio analysis model, the deep pronunciation features of the audio of the same audio category are subjected to compression mapping processing to obtain at least one compressed feature set.

[0093] Refer to Figure 6 , the deep pronunciation features of the audios belonging to the same audio category (label) l1 include n: and By performing compression mapping processing on the deep pronunciation features of the audios of the same audio category (label) l1, m (m << n) compressed features are obtained: The compressed feature set corresponding to an audio category (label) l1 composed of, and thus a total of C compressed feature sets are obtained.

[0094] Therefore, each compressed feature set corresponds to an audio category, each compressed feature set includes at least one compressed feature, and the feature dimension of each compressed feature set (i.e., the number of compressed features) is smaller than the deep pronunciation feature set under the audio category corresponding to each compressed feature set (a set of deep pronunciation features of audio of the same audio category).

[0095] Among them, when the deep pronunciation features of audio of the same audio category are compressed and mapped, in one embodiment, the deep pronunciation features of audio of the same audio category can be compressed and mapped by convolution processing to obtain a compressed feature set corresponding to the audio category. The convolution processing method is, for example, first constructing a feature matrix of the deep pronunciation features of audio of the same audio category, and then, the feature matrix can be convolved based on a preset convolution matrix to obtain a convolution result matrix, and each row in the convolution result matrix is ​​used as a compressed feature to obtain a compressed feature set. In another embodiment, the deep pronunciation features of audio of the same audio category can be compressed and mapped by long-term and short-term memory extraction compression. The long-term and short-term memory extraction compression method is, for example, aggregating part of the feature information of other deep pronunciation features for each deep pronunciation feature (for example, by dot multiplication, etc.) to obtain a compressed feature corresponding to each deep pronunciation feature, and then extracting part of the compressed features from all compressed features to obtain a compressed feature set. It can be understood that in other embodiments, compression mapping can be performed by compression implicit method.

[0096] By compressing the deep pronunciation feature set, a sparse pseudo sample (i.e., at least one compressed feature set) can be explicitly constructed with excellent audio information representation capabilities, so that the sparse Gaussian process in subsequent steps can reliably reduce the amount of calculation while ensuring the accuracy of audio category prediction.

[0097] In one embodiment, in step S330, the deep pronunciation features of the audios of the same audio category are compressed and mapped to obtain at least one compressed feature set, including:

[0098] According to the audio category corresponding to each audio, the deep pronunciation features of each audio are clustered to obtain at least one feature cluster, each feature cluster includes the deep pronunciation features of at least one audio, and each feature cluster corresponds to an audio category; each feature cluster is compressed and mapped into a corresponding compressed feature set to obtain at least one compressed feature set, and the feature dimension of each compressed feature set is smaller than the corresponding feature cluster.

[0099] According to the audio category corresponding to each audio, the deep pronunciation features of the audios of the same audio category can be aggregated together to obtain at least one feature cluster, for example, n deep pronunciation features of the audios belonging to the same audio category (label) l1: and A feature cluster can be formed. By performing compression mapping processing on the feature cluster, that is, for example, for and the formed feature cluster is subjected to compression mapping processing to obtain m (m << n) compressed features: the formed compressed feature set.

[0100] In one embodiment, each feature cluster is compressed and mapped to a corresponding compressed feature set, including:

[0101] Each feature cluster is respectively input into the compression mapping neural network; based on the compression mapping neural network, the input feature cluster is compressed and mapped into a corresponding compressed feature set.

[0102] The compression mapping neural network can be a convolutional neural network (which can perform convolutional processing), a long short-term memory network (which can perform long short-term memory extraction and compression processing), etc. The feature extraction neural network can be a superposition of multi-layer neural networks, or a superposition of multi-layer identical or different neural networks, such as a 3-layer convolutional network superposed as the compression mapping neural network. Based on the compression mapping neural network for the compression mapping neural network, the compression mapping neural network as a compression mapping neural network unit in the audio analysis model can further ensure the stability of the audio analysis model for audio category prediction.

[0103] In step S340, a Gaussian process is constructed based on at least one compressed feature set to obtain an audio category prediction function.

[0104] In the implementation manner of this example, the Gaussian process can be determined by the mean function m(x) and the covariance function (i.e., the Gaussian kernel) K. Among them, x is the compressed feature. The construction of the Gaussian process is the process of constructing the covariance function and the mean function based on at least one compressed feature set to obtain the audio category prediction function based on the mean function and the covariance function. The audio category prediction function obtained by constructing based on the Gaussian process is an audio category prediction function with parameters to be trained. The audio category prediction function in the trained audio analysis model can be used to predict the audio category of the audio to be classified.

[0105] Based on the input features (i.e., acoustic features) X = {x1,..., x N}, the construction of the Gaussian process can obtain the output f = {f(x1),..., f(x N ), where f(x) follows the Gaussian distribution N shown in formula (1). Among them, m(x) is the mean of the Gaussian distribution generated by operating on the input features based on the mean function, and K(X, X) is the covariance matrix generated by operating on the input features based on the covariance function (i.e., the Gaussian kernel).

[0106] f ∼ N(m(X), K(X, X)) (I)

[0107] For output Y, assuming that there is Gaussian noise between output Y and output f, the distribution of the final output y is shown in formula (2), where σ 2 I is a hyperparameter.

[0108] Y~N(m(X),K(X,X)+σ 2 I) (2)

[0109] Assuming that the joint probability distribution of the output Y (audio category) of the training data X based on the input features (i.e., acoustic features) and the output y* (audio category) of the test data x* based on the input features (i.e., acoustic features) conforms to the Gaussian distribution, we get formula (3), where the capital K is the covariance matrix corresponding to the test data, the lowercase k corresponds to the covariance matrix of the test data, and m(x) is set to 0, which can be set to the mean vector.

[0110]

[0111] For each audio category y* to be predicted, its posterior probability is shown in formula (4), and D can include the hyperparameter σ 2 I.

[0112]

[0113] Finally, we get the audio category prediction function As shown in formulas (5) and (6), is the predicted audio category, is the uncertainty score of the predicted audio category. A larger uncertainty score indicates a higher uncertainty of the predicted audio category.

[0114]

[0115]

[0116] The covariance function K can be selected according to the requirements, as shown in formula (7) in an example.

[0117]

[0118] Finally, in this application, based on at least one compressed feature set, the covariance function K(X, X) in the aforementioned Gaussian process is calculated as follows: i , x j |θ)→k(g(x i , w), g(x j, w)|θ, w) shown in the sparse Gaussian deep kernel learning, based on the deep feature extraction processing and compression mapping processing to obtain at least one compressed feature set as the input of the Gaussian process to construct the Gaussian process, the Gaussian kernel k(x) constructed based on the acoustic features i , x j |θ)(x i can be the i-th acoustic feature, x j can be the jth acoustic feature, θ can be a parameter in the covariance function), and is converted into a sparse Gaussian deep kernel k(g(x i , w), g(x j ,w)|θ,w)(g can represent deep feature extraction processing and compression mapping processing, and W can be a processing parameter), which can effectively reduce the computational complexity while ensuring the accuracy of audio category prediction.

[0119] In one embodiment, step S340, performing Gaussian process construction based on at least one compressed feature set to obtain an audio category prediction function, includes:

[0120] A training feature set and a test feature set are constructed based on compressed features in at least one compressed feature set; covariance operation is performed on the compressed features in the training feature set and the test feature set based on a covariance function to generate a target covariance matrix; mean operation is performed on the compressed features in the training feature set and the test feature set based on a mean function to generate a target mean vector; an audio category prediction function is constructed based on the covariance matrix and the target mean vector.

[0121] From the compressed features in at least one compressed feature set, a portion of the compressed features can be selected to construct a training feature set A part of the compressed features is selected to construct the test feature set h(x), and the dimensions of the compressed features in the training feature set and the test feature set can be selected according to the agreed dimensions.

[0122] The type of covariance function K can be selected according to the requirements. The covariance matrix can calculate the covariance between the compressed features to form a target covariance matrix. Each element in the target covariance matrix is ​​the covariance between the compressed features. The mean function m(x) can calculate the mean of the compressed features to obtain the target mean vector. Furthermore, based on the obtained covariance matrix and target mean vector, an audio category prediction function based on the parameters to be trained of the compressed feature set can be generated.

[0123] In one embodiment, covariance operation is performed on the compressed features in the training feature set and the test feature set based on the covariance function to generate a target covariance matrix, including:

[0124] Based on the covariance function, covariance operation is performed on the compressed features in the training feature set to obtain a first covariance matrix; based on the covariance function, covariance operation is performed on the compressed features in the test feature set to obtain a second covariance matrix; based on the covariance function, covariance operation is performed on the compressed features in the training feature set and the test feature set to obtain a third covariance matrix; the first covariance matrix, the second covariance matrix and the third covariance matrix are used as target covariance matrices.

[0125] Compress features in the training feature set Covariance calculations can be performed to generate compressed features that represent the training feature set. Pairwise covariance-based compression features The first covariance matrix of The first covariance matrix can transform K(X, X) in formula (3) into K(

[0126] Based on the covariance function, the covariance operation of the compressed features h(x) in the test feature set is performed to generate a second covariance matrix based on the compressed features h(x) that represents the covariance between each pair of compressed features h(x) in the test feature set. The second covariance matrix based on the compressed features h(x) can convert K(x*, x*) in formula (3) into K(h(x), h(x)).

[0127] Based on the covariance function, the covariance operation is performed on the compressed features in the training feature set and the test feature set to generate the compressed features h(x) in the test feature set and the compressed features in the training feature set. The covariance between them, based on the compressed features h(x) and The third covariance matrix of is based on the compressed features h(x) and The third covariance matrix can transform K(x*, X) in formula (3) into K(h(x), ).

[0128] In one embodiment, performing mean operation processing on the compressed features in the training feature set and the test feature set based on the mean function to generate a target mean vector includes:

[0129] Based on the mean function, the compressed features in the training feature set are processed by mean operation to obtain a first mean vector; based on the mean function, the compressed features in the training feature set are processed by mean operation to obtain a second mean vector; the first mean vector and the second mean vector are used as target mean vectors.

[0130] By taking the mean of all the compressed features h(x) in the test feature set, we can get the first mean vector μ1, and for all the compressed features in the training feature set By taking the mean, we can get the second mean vector μ2. Based on μ1 and μ2, we can generate a mean vector matrix to replace 0 in formula (3).

[0131] In one embodiment, constructing the audio category prediction function based on the covariance matrix and the target mean vector includes:

[0132] Obtain an audio category distribution function that satisfies a Gaussian distribution based on the mean function and the covariance function; obtain the audio category corresponding to the compressed feature in the training feature set; and construct an audio category prediction function that predicts the audio category corresponding to the compressed feature in the test feature set based on the audio category distribution function, the audio category corresponding to the compressed feature in the training feature set, the target covariance matrix, and the target mean vector.

[0133] The audio category distribution function may be a function satisfying Gaussian distribution N as shown in formula (1) generated according to a pre-selected mean function and covariance function. The audio category corresponding to the compressed feature set from which the compressed feature in the training feature set comes may be determined to obtain the audio category Y corresponding to the compressed feature in the training feature set.

[0134] Then, the audio category corresponding to the compressed features in the training feature set, the target covariance matrix and the target mean vector are input into the audio category distribution function, and then the audio category prediction function for predicting the audio category corresponding to the compressed features in the test feature set as shown in formulas (5) and (6) can be obtained. Then, the parameters in the audio analysis model can be adjusted according to the prediction results of the audio category prediction function to obtain the trained audio analysis model.

[0135] In step S350, the parameters in the audio analysis model are adjusted based on the audio category prediction function to obtain the audio category of the audio to be analyzed predicted by the trained audio analysis model.

[0136] In the implementation of this example, the audio category prediction function is an audio category prediction function with training parameters. The audio category prediction function is generated based on a compressed feature set, and can predict the audio category of a target compressed feature in the compressed feature set (e.g., a compressed feature in a test feature set) to obtain a prediction result. According to the error of the prediction result, the parameters in the audio analysis model can be adjusted until the prediction accuracy of the audio category prediction function meets the target requirements, thereby obtaining a trained audio analysis model. Based on the trained audio analysis model, the audio category of the audio to be analyzed can be predicted.

[0137] Among them, when adjusting the parameters in the audio analysis model, the adjusted parameters may include the parameters in the deep extraction processing process of the audio analysis model (such as parameters in the feature extraction neural network), the parameters in the compression mapping processing (such as parameters in the compression mapping neural network), and the parameters in the Gaussian process (such as parameters in the audio category prediction function, which may include hyperparameters in the covariance function) are jointly adjusted to form an overall sparse Gaussian deep kernel learning process, which effectively reduces the computational complexity and can ensure the accuracy of the audio analysis model for audio category prediction.

[0138] In one embodiment, in step S350, adjusting the parameters in the audio analysis model based on the audio category prediction function includes:

[0139] A target posterior probability of the audio category predicted by the audio category prediction function is determined; and a maximum likelihood estimation is performed based on the target posterior probability to adjust parameters in the audio analysis model.

[0140] The target posterior probability can be determined according to the audio category predicted by the audio category prediction function according to Bayesian estimation or other methods. The formula for determining the target posterior probability is shown in the following formula:

[0141]

[0142] Among them, T is transposed, Y is the audio category corresponding to the compressed feature in the training feature set, and y is the audio category corresponding to the compressed feature in the predicted test feature set. is the target posterior probability.

[0143] Then, the parameters in the audio analysis model can be updated and adjusted by performing gradient descent through maximum likelihood estimation based on the target posterior probability.

[0144] In one embodiment, determining a target posterior probability of the audio category predicted by the audio category prediction function includes:

[0145] Obtain a priori probability of the audio category predicted by the audio category prediction function; determine a posterior probability of the audio category predicted by the audio category prediction function; and use the product of the prior probability and the posterior probability as the target posterior probability.

[0146] The prior probability of the audio category predicted by the audio category prediction function may be the prior probability of the audio category corresponding to the compressed feature in the test feature set, for example The posterior probability of the audio class is The product of the prior probability and the posterior probability is

[0147] By combining the prior probability and the posterior probability in this way and then performing maximum likelihood estimation based on the parameters in the audio analysis model, the distribution of sparse pseudo samples (i.e., compressed feature sets) in the audio analysis model training process can be made closer to the distribution of actual deep pronunciation features, thereby further ensuring prediction accuracy.

[0148] Among them, by combining the prior probability and the posterior probability to obtain the target posterior probability, the maximum likelihood estimation can be performed according to the following likelihood estimation formula:

[0149]

[0150] In the likelihood estimation formula, θ can be a parameter in the depth extraction process and a parameter in the compression mapping process, σ y , l can be a parameter in the Gaussian process. The likelihood estimation formula can be decomposed into two loss functions loss1 and loss2, as shown in the following formula:

[0151]

[0152] Among them, reference Figure 7 As shown, the loss function loss1 can be based on the compression feature h(x) (including h(x1) to h(x n )) The corresponding prediction error estimate, loss function loss2 can be expressed based on the compressed features in the training feature set (include to ) is an estimate of the prediction error corresponding to .

[0153] In one application scenario, the audio in the aforementioned embodiment is spoken audio, and the audio to be analyzed includes spoken audio of following the target text; obtaining the audio category of the audio to be analyzed predicted by the trained audio analysis model includes:

[0154] Based on the target text, acoustic feature information and phoneme alignment information corresponding to the spoken audio are extracted; the acoustic feature information and phoneme alignment information corresponding to the spoken audio are input into the trained audio analysis model to output the audio category of the spoken audio.

[0155] Follow the spoken audio of the target text, for example, the spoken audio generated by the user following the English text in the English learning application, and then, based on the spoken audio, i.e., the target text, according to the embodiment corresponding to step S310, extract the acoustic feature information and phoneme alignment information corresponding to the spoken audio. Input the acoustic feature information and phoneme alignment information corresponding to the spoken audio into the trained audio analysis model, and the deep pronunciation features corresponding to the spoken audio can be extracted based on the trained deep extraction process in step S320. Then, based on the trained compression mapping process in step S330, the deep pronunciation features corresponding to the spoken audio are compressed and mapped to obtain the compressed features corresponding to the spoken audio, and finally, the compressed features are predicted based on the trained audio category prediction function in step S330 to obtain the audio category corresponding to the spoken audio, and the uncertainty corresponding to the audio category can be output at the same time.

[0156] See also Figures 8 to 10 The terminal interface changes shown in Figure 8 In the terminal interface shown, the user can trigger (such as clicking or long pressing) to "start reading" and then read the text "ilike apple". Figure 1 The terminal 102 shown or Figure 2 The client 201 shown corresponds to the terminal) can collect the user's spoken audio, and then Fig. 9 In the terminal interface shown, the user can trigger (for example, click once or release the long press) "end reading" to make the terminal complete the collection of the user's spoken audio. At this time, the terminal can send the spoken audio and text to the server (for example Figure 1 The server 101 shown or Figure 2 The node 202 shown corresponds to the server), the server can predict the audio category and the uncertainty score of the spoken audio based on the spoken audio and text based on the trained audio analysis model, and finally, through Fig.10 The terminal interface shown shows that the audio category is four stars, and the uncertainty score (ie, score confidence) of four stars is 0.5.

[0157] In this way, based on steps S310 to S350, at least one compressed feature set is obtained based on the deep feature extraction processing and the compression mapping processing as the input of the Gaussian process to construct the Gaussian process, and the relationship between the acoustic feature information and the pseudo sample (ie, the compressed feature set) is explicitly modeled, and a sparse Gaussian deep kernel learning process is established, which can effectively reduce the computational complexity while ensuring the accuracy of audio category prediction.

[0158] In order to facilitate better implementation of the audio category prediction method provided in the embodiment of the present application, the embodiment of the present application also provides an audio category prediction device based on the above audio category prediction method. The meanings of the terms are the same as those in the above audio category prediction method, and the specific implementation details can refer to the description in the method embodiment. Fig.11 A block diagram of an audio category prediction device according to an embodiment of the present application is shown. Fig.11 A block diagram of an audio category prediction device according to another embodiment of the present application is shown.

[0159] like Fig.11 As shown, the audio category prediction device 400 may include an acquisition module 410, an input module 420, a compression module 430, a Gaussian module 440 and a prediction module 450, and the audio category prediction device 400 may be applied to a terminal.

[0160] The acquisition module 410 can be used to acquire the acoustic feature information and phoneme alignment information of at least one audio, and each audio is calibrated to a corresponding audio category; the input module 420 can be used to use an audio analysis model to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each audio, and obtain the deep pronunciation features of each audio; the compression module 430 can be used to compress and map the deep pronunciation features of the audio of the same audio category, and obtain at least one compressed feature set; the Gaussian module 440 can be used to construct a Gaussian process based on the at least one compressed feature set to obtain an audio category prediction function; the prediction module 450 can be used to adjust the parameters in the audio analysis model based on the audio category prediction function, so as to obtain the trained audio analysis model to predict the audio category of the audio to be analyzed.

[0161] In some embodiments of the present application, the acoustic feature information of the audio includes at least one frame of sub-acoustic features of the audio, and the phoneme alignment information of the audio includes the pronunciation start and end time periods of the words in the audio; the input module includes: a depth extraction unit, which is used to perform deep extraction processing on at least one frame of sub-acoustic features of each of the audios to obtain at least one frame of deep features corresponding to each of the audios; a first normalization unit, which is used to normalize the depth features of at least one frame of deep features corresponding to each of the audios that belong to the same pronunciation start and end time period to obtain the word features of each word in the audio; a second normalization unit, which is used to normalize the word features of each word in the audio to obtain the deep pronunciation features of each of the audios.

[0162] In some embodiments of the present application, the deep extraction unit includes: a network input subunit, used to input at least one frame of sub-acoustic features of each of the audio into a feature extraction neural network; a network extraction subunit, used to perform deep extraction processing on the input at least one frame of sub-acoustic features based on the feature extraction neural network to obtain at least one frame of deep features corresponding to each of the audio.

[0163] In some embodiments of the present application, the compression module includes: a clustering unit, which is used to cluster the deep pronunciation features of each audio according to the audio category corresponding to each audio, so as to obtain at least one feature cluster, each of which includes the deep pronunciation features of at least one audio, and each of which corresponds to an audio category; a mapping unit, which is used to compress and map each of the feature clusters into a corresponding compressed feature set, so as to obtain at least one compressed feature set, and the feature dimension of each of the compressed feature sets is smaller than that of the corresponding feature cluster.

[0164] In some embodiments of the present application, the mapping unit includes: a mapping input subunit, used to input each of the feature clusters into a compression mapping neural network respectively; and a network implicit unit, used to compress and map the input feature cluster into a corresponding compression feature set based on the compression mapping neural network.

[0165] In some embodiments of the present application, the Gaussian module includes: a sample construction module, which is used to construct a training feature set and a test feature set based on the compressed features in the at least one compressed feature set; a matrix generation unit, which is used to perform covariance operation processing on the compressed features in the training feature set and the test feature set based on the covariance function to generate a target covariance matrix; a mean generation unit, which is used to perform mean operation processing on the compressed features in the training feature set and the test feature set based on the mean function to generate a target mean vector; and a function construction unit, which is used to construct the audio category prediction function based on the covariance matrix and the target mean vector.

[0166] In some embodiments of the present application, the matrix generation unit includes: a first covariance generation subunit, used to perform covariance operation processing on the compressed features in the training feature set based on the covariance function to obtain a first covariance matrix; a second covariance generation subunit, used to perform covariance operation processing on the compressed features in the test feature set based on the covariance function to obtain a second covariance matrix; a third generation subunit, used to perform covariance operation processing on the compressed features in the training feature set and the test feature set based on the covariance function to obtain a third covariance covariance matrix; a matrix determination subunit, used to use the first covariance matrix, the second covariance matrix and the third covariance matrix as the target covariance matrix.

[0167] In some embodiments of the present application, the mean generation unit includes: a first mean generation subunit, used to perform mean operation processing on the compressed features in the training feature set based on the mean function to obtain a first mean vector; a second mean generation subunit, used to perform mean operation processing on the compressed features in the training feature set based on the mean function to obtain a second mean vector; and a mean determination subunit, used to use the first mean vector and the second mean vector as the target mean vector.

[0168] In some embodiments of the present application, the function construction unit includes: a distribution function acquisition subunit, used to obtain an audio category distribution function that satisfies a Gaussian distribution and is generated based on the mean function and the covariance function; a training category acquisition subunit, used to obtain the audio category corresponding to the compressed feature in the training feature set; and a prediction function construction subunit, used to construct an audio category prediction function that predicts the audio category corresponding to the compressed feature in the test feature set based on the audio category distribution function, the audio category corresponding to the compressed feature in the training feature set, the target covariance matrix and the target mean vector.

[0169] In some embodiments of the present application, the prediction module includes: a target posterior probability determination unit, used to determine the target posterior probability of the audio category predicted by the audio category prediction function; and an estimation unit, used to perform maximum likelihood estimation based on the target posterior probability to adjust the parameters in the audio analysis model.

[0170] In some embodiments of the present application, the target posterior probability determination unit includes: a first probability acquisition subunit, used to obtain the prior probability of the audio category predicted by the audio category prediction function; a second probability acquisition subunit, used to determine the posterior probability of the audio category predicted by the audio category prediction function; and a third probability acquisition subunit, used to take the product of the prior probability and the posterior probability as the target posterior probability.

[0171] In some embodiments of the present application, the audio to be analyzed includes spoken audio that follows a target text; the prediction module includes: a spoken information extraction unit, used to extract acoustic feature information and phoneme alignment information corresponding to the spoken audio based on the target text; a spoken classification unit, used to input the acoustic feature information and phoneme alignment information corresponding to the spoken audio into the trained audio analysis model to output the audio category of the spoken audio.

[0172] In this way, based on the audio category prediction device 400, at least one compressed feature set can be obtained based on deep feature extraction processing and compression mapping processing as the input of the Gaussian process to construct the Gaussian process, explicitly model the relationship between the acoustic feature information and the pseudo sample (i.e., the compressed feature set), and establish a sparse Gaussian deep kernel learning process, which can effectively reduce the computational complexity while ensuring the accuracy of audio category prediction.

[0173] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0174] In addition, an embodiment of the present application further provides an electronic device, which may be a terminal or a server. Fig.12 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0175] The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will appreciate that Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0176] The processor 501 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 502 and calling data stored in the memory 502, it executes various functions of the computer device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501.

[0177] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0178] The electronic device also includes a power supply 503 for supplying power to each component. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 503 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0179] The electronic device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0180] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 501 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 will run the application programs stored in the memory 502, thereby realizing various functions, such as the processor 501 can execute:

[0181] Acquiring acoustic feature information and phoneme alignment information of at least one audio, and calibrating the corresponding audio category of each audio; using an audio analysis model, performing deep extraction processing on the acoustic feature information and phoneme alignment information of each audio, and obtaining deep pronunciation features of each audio; performing compression mapping processing on the deep pronunciation features of the audio of the same audio category, and obtaining at least one compressed feature set; constructing a Gaussian process based on the at least one compressed feature set to obtain an audio category prediction function; adjusting the parameters in the audio analysis model based on the audio category prediction function to obtain the audio category of the audio to be analyzed predicted by the trained audio analysis model.

[0182] In one embodiment, the acoustic feature information of the audio includes at least one frame of sub-acoustic features of the audio, and the phoneme alignment information of the audio includes the pronunciation start and end time periods of words in the audio; the deep extraction processing of the acoustic feature information and phoneme alignment information of each audio to obtain the deep pronunciation features of each audio includes: deep extraction processing of at least one frame of sub-acoustic features of each audio to obtain at least one frame of deep features corresponding to each audio; normalizing the deep features of at least one frame of deep features corresponding to each audio that belong to the same pronunciation start and end time period to obtain the word features of each word in the audio; normalizing the word features of each word in the audio to obtain the deep pronunciation features of each audio.

[0183] In one embodiment, the deep extraction processing of at least one frame of sub-acoustic features of each of the audios to obtain at least one frame of deep features corresponding to each of the audios includes: inputting at least one frame of sub-acoustic features of each of the audios into a feature extraction neural network; and deep extraction processing of the input at least one frame of sub-acoustic features based on the feature extraction neural network to obtain at least one frame of deep features corresponding to each of the audios.

[0184] In one embodiment, the deep pronunciation features of the audios of the same audio category are compressed and mapped to obtain at least one compressed feature set, including: clustering the deep pronunciation features of each audio according to the audio category corresponding to each audio to obtain at least one feature cluster, each feature cluster including the deep pronunciation features of at least one audio, and each feature cluster corresponding to an audio category; compressing and mapping each feature cluster into a corresponding compressed feature set to obtain the at least one compressed feature set, and the feature dimension of each compressed feature set is smaller than the corresponding feature cluster.

[0185] In one embodiment, compressing and mapping each of the feature clusters into a corresponding compressed feature set includes: inputting each of the feature clusters into a compression mapping neural network respectively; and compressing and mapping the input feature clusters into a corresponding compressed feature set based on the compression mapping neural network.

[0186] In one embodiment, the Gaussian process is constructed based on the at least one compressed feature set to obtain the audio category prediction function, including: constructing a training feature set and a test feature set based on the compressed features in the at least one compressed feature set; performing covariance operation processing on the compressed features in the training feature set and the test feature set based on the covariance function to generate a target covariance matrix; performing mean operation processing on the compressed features in the training feature set and the test feature set based on the mean function to generate a target mean vector; and constructing the audio category prediction function based on the covariance matrix and the target mean vector.

[0187] In one embodiment, the covariance operation is performed on the compressed features in the training feature set and the test feature set based on the covariance function to generate a target covariance matrix, including: performing covariance operation on the compressed features in the training feature set based on the covariance function to obtain a first covariance matrix; performing covariance operation on the compressed features in the test feature set based on the covariance function to obtain a second covariance matrix; performing covariance operation on the compressed features in the training feature set and the test feature set based on the covariance function to obtain a third covariance matrix; and using the first covariance matrix, the second covariance matrix and the third covariance matrix as the target covariance matrix.

[0188] In one embodiment, the mean operation is performed on the compressed features in the training feature set and the test feature set based on the mean function to generate a target mean vector, including: performing mean operation on the compressed features in the training feature set based on the mean function to obtain a first mean vector; performing mean operation on the compressed features in the training feature set based on the mean function to obtain a second mean vector; and using the first mean vector and the second mean vector as the target mean vector.

[0189] In one embodiment, constructing the audio category prediction function based on the covariance matrix and the target mean vector includes: obtaining an audio category distribution function that satisfies a Gaussian distribution and is generated based on the mean function and the covariance function; obtaining the audio category corresponding to the compressed feature in the training feature set; and constructing an audio category prediction function that predicts the audio category corresponding to the compressed feature in the test feature set based on the audio category distribution function, the audio category corresponding to the compressed feature in the training feature set, the target covariance matrix and the target mean vector.

[0190] In one embodiment, the adjusting the parameters in the audio analysis model based on the audio category prediction function includes: determining a target posterior probability of the audio category predicted by the audio category prediction function; and performing maximum likelihood estimation based on the target posterior probability to adjust the parameters in the audio analysis model.

[0191] In one embodiment, determining the target posterior probability of the audio category predicted by the audio category prediction function includes: obtaining the prior probability of the audio category predicted by the audio category prediction function; determining the posterior probability of the audio category predicted by the audio category prediction function; and taking the product of the prior probability and the posterior probability as the target posterior probability.

[0192] In one embodiment, the audio to be analyzed includes spoken audio that follows a target text; obtaining a trained audio analysis model to predict the audio category of the audio to be analyzed includes: extracting acoustic feature information and phoneme alignment information corresponding to the spoken audio based on the target text; inputting the acoustic feature information and phoneme alignment information corresponding to the spoken audio into the trained audio analysis model to output the audio category of the spoken audio.

[0193] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0194] To this end, an embodiment of the present application further provides a storage medium, in which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any method provided in the embodiment of the present application.

[0195] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0196] Since the computer program stored in the storage medium can execute the steps in any method provided in the embodiments of the present application, the beneficial effects that can be achieved by the method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0197] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations in the above embodiments of the present application.

[0198] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0199] It should be understood that the present application is not limited to the embodiments that have been described above and shown in the accompanying drawings, but various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for predicting audio categories, characterized in that: include: Acquire acoustic feature information and phoneme alignment information of at least one audio, and calibrate the audio category corresponding to each of the audio; Using an audio analysis model, deep extraction and processing are performed on the acoustic feature information and phoneme alignment information of each of the audios to obtain deep pronunciation features of each of the audios; Performing compression mapping processing on the deep pronunciation features of the audio of the same audio category to obtain at least one compressed feature set; Performing Gaussian process construction based on the at least one compressed feature set to obtain an audio category prediction function; The parameters in the audio analysis model are adjusted based on the audio category prediction function to obtain the audio category of the audio to be analyzed predicted by the trained audio analysis model.

2. The method according to claim 1, characterized in that The acoustic feature information of the audio includes at least one sub-frame acoustic feature of the audio, and the phoneme alignment information of the audio includes the pronunciation start and end time periods of words in the audio; The deep extraction process of the acoustic feature information and the phoneme alignment information of each audio to obtain the deep pronunciation feature of each audio includes: Performing a deep extraction process on at least one frame of sub-acoustic features of each of the audios to obtain at least one frame of deep features corresponding to each of the audios; Normalizing the depth features of at least one frame of depth features corresponding to each of the audios that belong to the same pronunciation start and end time period to obtain word features of each word in the audios; The word features of each word in the audio are normalized to obtain the deep pronunciation features of each audio.

3. The method according to claim 2, characterized in that The performing deep extraction processing on at least one frame of sub-acoustic features of each of the audios to obtain at least one frame of deep features corresponding to each of the audios includes: Inputting at least one frame of sub-acoustic features of each of the audio into a feature extraction neural network; Based on the feature extraction neural network, deep extraction processing is performed on at least one frame of sub-acoustic features input to obtain at least one frame of deep features corresponding to each of the audio.

4. The method according to claim 1, characterized in that: The deep pronunciation features of the audio of the same audio category are compressed and mapped to obtain at least one compressed feature set, including: According to the audio category corresponding to each of the audios, clustering processing is performed on the deep pronunciation features of each of the audios to obtain at least one feature cluster, each of the feature clusters includes the deep pronunciation features of at least one audio, and each of the feature clusters corresponds to an audio category; Each of the feature clusters is compressed and mapped into a corresponding compressed feature set to obtain the at least one compressed feature set, wherein the feature dimension of each compressed feature set is smaller than the corresponding feature cluster.

5. The method according to claim 4, characterized in that The compressing and mapping each of the feature clusters into a corresponding compressed feature set comprises: Input each of the feature clusters into a compression mapping neural network respectively; Based on the compression mapping neural network, the input feature cluster is compressed and mapped into a corresponding compression feature set.

6. The method according to claim 1, characterized in that The performing Gaussian process construction based on the at least one compressed feature set to obtain an audio category prediction function comprises: Constructing a training feature set and a test feature set based on the compressed features in the at least one compressed feature set; Based on the covariance function, covariance operation is performed on the compressed features in the training feature set and the test feature set to generate a target covariance matrix; Performing mean operation processing on the compressed features in the training feature set and the test feature set based on the mean function to generate a target mean vector; The audio category prediction function is constructed based on the covariance matrix and the target mean vector.

7. The method according to claim 6, characterized in that The covariance operation is performed on the compressed features in the training feature set and the test feature set based on the covariance function to generate a target covariance matrix, including: Performing covariance operation processing on the compressed features in the training feature set based on the covariance function to obtain a first covariance matrix; Based on the covariance function, a covariance operation is performed on the compressed features in the test feature set to obtain a second covariance matrix; Performing covariance operation processing on the compressed features in the training feature set and the test feature set based on the covariance function to obtain a third covariance matrix; The first covariance matrix, the second covariance matrix and the third covariance matrix are used as the target covariance matrix.

8. The method according to claim 6, characterized in that The performing mean operation processing on the compressed features in the training feature set and the test feature set based on the mean function to generate a target mean vector includes: Performing mean operation processing on the compressed features in the training feature set based on the mean function to obtain a first mean vector; Performing mean operation processing on the compressed features in the training feature set based on the mean function to obtain a second mean vector; The first mean vector and the second mean vector are used as the target mean vector.

9. The method according to claim 6, characterized in that The step of constructing the audio category prediction function based on the covariance matrix and the target mean vector includes: Obtaining an audio category distribution function that satisfies a Gaussian distribution and is generated based on the mean function and the covariance function; Obtaining the audio category corresponding to the compressed feature in the training feature set; An audio category prediction function is constructed based on the audio category distribution function, the audio category corresponding to the compressed features in the training feature set, the target covariance matrix, and the target mean vector to predict the audio category corresponding to the compressed features in the test feature set.

10. The method according to claim 1, characterized in that The adjusting the parameters in the audio analysis model based on the audio category prediction function includes: Determining a target posterior probability of the audio category predicted by the audio category prediction function; Maximum likelihood estimation is performed based on the target posterior probability to adjust parameters in the audio analysis model.

11. The method according to claim 10, characterized in that The determining a target posterior probability of the audio category predicted by the audio category prediction function comprises: Obtaining a priori probability of the audio category predicted by the audio category prediction function; Determining a posterior probability of the audio category predicted by the audio category prediction function; The product of the prior probability and the posterior probability is used as the target posterior probability.

12. The method according to claim 1, characterized in that The audio to be analyzed includes spoken audio of the target text; The step of obtaining the audio category of the audio to be analyzed predicted by the trained audio analysis model includes: Extracting acoustic feature information and phoneme alignment information corresponding to the spoken audio based on the target text; The acoustic feature information and phoneme alignment information corresponding to the spoken audio are input into the trained audio analysis model to output the audio category of the spoken audio.

13. An audio category prediction device, characterized in that: include: An acquisition module, used to acquire acoustic feature information and phoneme alignment information of at least one audio, each audio calibrated to a corresponding audio category; An input module, used to use an audio analysis model to perform deep extraction processing on the acoustic feature information and phoneme alignment information of each of the audios to obtain deep pronunciation features of each of the audios; A compression module, used for compressing and mapping the deep pronunciation features of the audio of the same audio category to obtain at least one compressed feature set; A Gaussian module, configured to perform Gaussian process construction based on the at least one compressed feature set to obtain an audio category prediction function; A prediction module is used to adjust the parameters in the audio analysis model based on the audio category prediction function to obtain the audio category of the audio to be analyzed predicted by the trained audio analysis model.

14. A storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 12.

15. An electronic device, characterized in that: include: a memory storing computer-readable instructions; A processor reads the computer-readable instructions stored in the memory to execute the method according to any one of claims 1 to 12.

Images