Audio processing methods, devices, electronic equipment and storage media

Abstract

This application discloses an audio processing method, apparatus, electronic device, and storage medium, belonging to the field of audio technology. This application achieves this by performing only one intersection operation on the predicted word lattice and the word lattice to be processed. Utilizing the target word lattice obtained from this single intersection operation and the path relationship information as its byproduct, a prediction score can be assigned to each state in the target word lattice. This allows for convenient frame-by-frame reconstruction of the prediction probability of candidate text using the prediction scores of each state. This enables the selection of the recognition text with the highest semantic matching degree based on the prediction probability. Thus, without sacrificing audio decoding accuracy, the single intersection operation significantly reduces computational overhead, thereby greatly saving computational resources, increasing computational speed, and ultimately improving audio decoding speed.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to an audio processing method, apparatus, electronic device and storage medium. Background Technology

[0002] In audio recognition systems, audio recognition tasks are typically divided into a training phase and a decoding phase. In the training phase, labeled audio-text pairs are used to train an ASR (Automatic Speech Recognition) neural network. In the decoding phase, the ASR neural network is used to decode the input audio to obtain the text corresponding to the phonemes contained in the audio.

[0003] Currently, the LF-MMI (Lattice-Free Maximum Mutual Information) method can be used both to train ASR neural networks and to decode input audio. However, decoding using LF-MMI requires a frame-by-frame intersection operation between the ASR neural network output distribution and the input numerator / denominator FST (Finite-State Transducer). This decoding process is computationally expensive and slow. Summary of the Invention

[0004] This application provides an audio processing method, apparatus, electronic device, and storage medium that can improve audio decoding speed by saving computational resources and increasing computational speed without sacrificing audio decoding accuracy. The technical solution is as follows:

[0005] On the one hand, an audio processing method is provided, the method comprising:

[0006] Decode multiple audio frames in the audio to be identified to obtain a predicted word grid for the multiple audio frames. The predicted word grid provides a character sequence of multiple candidate texts identified from the multiple audio frames.

[0007] Based on any candidate text from the plurality of audio frames, a word grid to be operated on is obtained, wherein the word grid to be operated on is used to provide the character sequence of the candidate text or the global character sequence of different candidate texts;

[0008] An intersection operation is performed on the predicted word grid and the word grid to be operated on to obtain the target word grid and path relationship information. The path relationship information indicates the association relationship between each arc in the target word grid and the same arc in the word grid to be operated on. Each arc in the target word grid indicates a character.

[0009] Based on the target word grid and the path relationship information, the prediction scores of multiple states in the target word grid are determined, and the prediction scores indicate the degree of semantic matching between each state and the preceding audio frame in the audio to be identified;

[0010] Based on the prediction scores of multiple states in the target word lattice, the prediction probability of the candidate text is determined, and the prediction probability characterizes the degree of semantic matching between the candidate text and the audio to be identified.

[0011] On one hand, an audio processing apparatus is provided, the apparatus comprising:

[0012] The decoding module is used to decode multiple audio frames in the audio to be identified, and obtain the predicted word grids of the multiple audio frames. The predicted word grids provide character sequences of multiple candidate texts identified from the multiple audio frames.

[0013] The acquisition module is used to acquire a word grid to be operated on based on any candidate text of the plurality of audio frames. The word grid to be operated on is used to provide the character sequence of the candidate text or the global character sequence of different candidate texts.

[0014] The intersection operation module is used to perform an intersection operation on the predicted word grid and the word grid to be operated on to obtain the target word grid and path relationship information. The path relationship information indicates the association relationship between each arc in the target word grid and the same arc in the word grid to be operated on. Each arc in the target word grid indicates a state.

[0015] The score determination module is used to determine the predicted scores of multiple states in the target word grid based on the target word grid and the path relationship information. The predicted scores indicate the degree of semantic matching between each state and the preceding audio frame in the audio to be identified.

[0016] The probability determination module is used to determine the prediction probability of the candidate text based on the prediction scores of multiple characters in the target word grid. The prediction probability represents the degree of semantic matching between the candidate text and the audio to be identified.

[0017] In some embodiments, the score determination module includes:

[0018] The filtering unit is used to filter each arc in the target word grid to obtain multiple target arcs, so that each state in the target word grid has one and only one target arc that reaches the state;

[0019] An arc determination unit is used to determine, based on the path relationship information, the associated arc of each target arc from the word grid to be operated on, wherein the target arc and the associated arc indicate the same character in the word grid;

[0020] The assignment unit is used to assign a value to the predicted score of the arrival state in the target word lattice based on the forward score of the arrival state connected by each associated arc in the word lattice to be operated.

[0021] In some embodiments, the score determination module further includes:

[0022] A state determination unit is used to determine the arrival state connected to each associated arc in the word lattice to be operated on;

[0023] The score acquisition unit is used to acquire the forward score of the arrival state based on each arc in the word lattice that reaches the arrival state.

[0024] In some embodiments, the score acquisition unit is used for:

[0025] For each arc in the word lattice that reaches the arrival state, the forward fraction of the starting state connected to each arc and the weight on each arc are added together to obtain the forward weight parameter of each arc.

[0026] Using the natural base as the base and the forward weight parameter of each arc as the exponent, obtain the forward weight index of each arc;

[0027] The forward score of the following character is obtained by taking the logarithm of the sum of the forward weight indices of each arc.

[0028] In some embodiments, the assignment unit is used for:

[0029] The query is performed on each state of the word to be operated on to obtain multiple candidate states that meet the query conditions. The query conditions indicate that there is an arc that starts from the current state and reaches the final state. The arc that reaches the final state indicates the character identified in the last audio frame among the multiple audio frames.

[0030] If the arrival state belongs to the candidate state, the weight on the arc from the arrival state to the terminal character is added to the forward score of the arrival state to obtain the predicted score of the arrival state.

[0031] If the arrival state does not belong to the candidate state, a preset value is assigned to the predicted score of the arrival state.

[0032] In some embodiments, the word grid to be operated on includes a numerator word grid and a denominator word grid, wherein the numerator word grid provides the character sequence of the candidate text and the denominator word grid provides the global character sequence of different candidate texts;

[0033] The intersection operation module is used for:

[0034] Perform an intersection operation on the predicted word lattice and the molecular word lattice to obtain the target molecular word lattice;

[0035] The target denominator lattice is obtained by performing an intersection operation on the predicted lattice and the denominator lattice.

[0036] In some embodiments, the probability determination module includes:

[0037] The first acquisition unit is used to acquire the molecular prediction component of each audio frame in the plurality of audio frames based on the target molecular word lattice.

[0038] The second acquisition unit is used to acquire the denominator prediction component of the audio frame based on the target denominator word lattice;

[0039] The prediction component determination unit is used to determine the single-frame prediction component of the audio frame based on the numerator prediction component and the denominator prediction component of the audio frame.

[0040] The probability acquisition unit is used to acquire the prediction probability of the candidate text based on the single-frame prediction components of each of the multiple audio frames.

[0041] In some embodiments, the first acquisition unit is used to:

[0042] From the target molecular lattice, query at least one candidate molecular state identified from the audio frame;

[0043] Using the natural base as the base and the prediction score of each candidate molecule state as the index, the prediction index of each candidate molecule state is obtained.

[0044] The molecular prediction component of the audio frame is obtained by taking the logarithm of the sum of the prediction indices of the at least one candidate molecular state.

[0045] In some embodiments, the second acquisition unit is used for:

[0046] From the target denominator lattice, query at least one candidate denominator state obtained from the audio frame;

[0047] Using the natural base as the base and the predicted score of each candidate denominator state as the exponent, obtain the predicted exponent for each candidate denominator state.

[0048] The denominator prediction component of the audio frame is obtained by taking the logarithm of the sum of the prediction exponents of the at least one candidate denominator state.

[0049] On one hand, an electronic device is provided, comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, the at least one computer program being loaded and executed by the one or more processors to implement the audio processing method described above.

[0050] On one hand, a computer-readable storage medium is provided, which stores at least one computer program that is loaded and executed by a processor to implement the audio processing method described above.

[0051] On one hand, a computer program product is provided, comprising one or more computer programs stored in a computer-readable storage medium. One or more processors of an electronic device are capable of reading the one or more computer programs from the computer-readable storage medium, and the one or more processors execute the one or more computer programs, enabling the electronic device to perform the aforementioned audio processing method.

[0052] The beneficial effects of the technical solutions provided in this application include at least the following:

[0053] By performing only one intersection operation on the predicted word lattice and the word lattice to be operated on, and using the target word lattice obtained from the intersection operation and the path relationship information as its byproduct, a prediction score can be assigned to each state in the target word lattice. This allows the prediction probability of the candidate text to be easily reconstructed frame by frame using the prediction scores of each state. This enables the selection of the recognition text with the highest semantic matching degree based on the prediction probability. In this way, without sacrificing audio decoding accuracy, the computational overhead is greatly compressed by the single intersection operation, thereby greatly saving computing resources and improving computing speed, thus improving audio decoding speed. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of a different type of FST provided in the embodiments of this application;

[0056] Figure 2 This is a schematic diagram of an intersection operation provided in an embodiment of this application;

[0057] Figure 3This is a schematic diagram of the implementation environment of an audio processing method provided in an embodiment of this application;

[0058] Figure 4 This is a flowchart of an audio processing method provided in an embodiment of this application;

[0059] Figure 5 This is a flowchart of an audio processing method provided in an embodiment of this application;

[0060] Figure 6 This is a schematic diagram of a prediction lattice nnet_output provided in an embodiment of this application;

[0061] Figure 7 This is a schematic diagram of a lattice a_fsa to be operated on, provided in an embodiment of this application;

[0062] Figure 8 This is a schematic diagram of a target lattice provided in an embodiment of this application;

[0063] Figure 9 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application;

[0064] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0065] Figure 11 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0067] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0068] In this application, the term "at least one" means one or more, and "multiple" means two or more, for example, multiple first positions means two or more first positions.

[0069] The term "including at least one of A or B" in this application refers to the following situations: including only A, including only B, and including both A and B.

[0070] The user-related information (including but not limited to user device information, personal information, behavioral information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application, when applied to specific products or technologies using the methods of the embodiments of this application, are all obtained with the user's permission, consent, authorization, or full authorization from all parties. Furthermore, the collection, use, and processing of related information, data, and signals must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the audio to be identified involved in this application was obtained under fully authorized conditions.

[0071] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0072] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0073] Key technologies in speech technology include automatic speech recognition and speech synthesis. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech being one of the most promising methods for future human-computer interaction.

[0074] Automatic speech recognition (ASR) is a technology that converts human speech into text. Speech recognition is a multidisciplinary field, closely linked to acoustics, phonetics, linguistics, digital signal processing theory, information theory, computer science, and many other disciplines. Due to the diversity and complexity of speech signals, speech recognition systems can only achieve satisfactory performance under certain constraints, or in other words, can only be applied to specific situations.

[0075] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0076] The solutions provided in this application involve AI-based ASR technology. Hereinafter, some basic concepts of ASR technology will be explained and described.

[0077] ASR system: refers to an automatic speech recognition system. When an audio segment is input into an ASR system, it can automatically convert the audio into semantically corresponding text, thus achieving automatic audio-to-text conversion. The operation of an ASR system is generally divided into a training phase and a decoding phase. In the training phase, the ASR system is built using labeled audio-text pairs. In the decoding phase, the trained ASR system is used to automatically recognize the audio to be recognized.

[0078] Discriminative Training: In an ASR system, given an acoustic feature sequence O, the predicted probabilities of character sequences W for multiple candidate texts are calculated. The candidate text with the highest predicted probability is the most likely recognition result; that is, the candidate text with the highest predicted probability is output as the final recognized text for the input audio. ASR neural network training is generally based on the Maximum Likelihood (ML) criterion. However, the ML criterion only considers optimizing the correct path and does not consider reducing the predicted probabilities of other paths, thus resulting in poor recognition performance. The goal of discriminative training is to increase the predicted probability of the correct path while reducing the predicted probabilities of other paths, thereby increasing the differences between different paths and thus achieving better recognition results. In contrast to the ML criterion, the criterion for discriminative training is called the Maximum Mutual Information (MMI) criterion.

[0079] The LF-MMI (Lattice-Free Maximum Mutual Information) criterion is a training criterion applicable to ASR systems, and can be considered a specific engineering implementation based on the MMI criterion. The calculation of the LF-MMI criterion relies on related algorithms of FST (Finite-State Transducer). The LF-MMI criterion is not only applicable to the training phase of ASR systems, but the technical solutions provided in this application embodiment can also be applied to the decoding phase of ASR systems.

[0080] FST (Finite-State Transducer): An abstract mathematical model widely used in speech recognition, natural language search, and processing. In the embodiments of this application, the word lattices involved in the ASR system all belong to FST.

[0081] Intersection operation: An operation applied to two FSTs, producing a new FST that contains all intersection paths of the two FSTs. In this embodiment, the intersection operation on two word lattices can extract all intersection character sequences from the two word lattices.

[0082] The following section will provide a detailed introduction to the LF-MMI criterion and its related algorithms based on FST.

[0083] 1. FST's state and arc

[0084] FST is an abstract mathematical model widely used in ASR systems. FST mainly consists of states and arcs: the states of FST are divided into initial states, terminal states, and ordinary states. The starting point of each path in FST can be called the initial state, and the ending point of each path can be called the terminal state. The remaining states are ordinary states, which are states that are neither initial states nor terminal states. The directed edge connecting two states in FST is called an arc of FST. Each arc of FST has at least one input label.

[0085] When the initial state, the final state, and each arc in an FST have weights, the FST is called a weighted finite-state transducer (WFST). When there are output labels on the arcs of the FST, it is called a transducer; otherwise, when there are no output labels on the arcs of the FST, it is called an acceptor.

[0086] Based on whether the FST is weighted and whether it carries an output label on the arc, FSTs can be divided into the following four categories: FSA (Finite-State Acceptor), WFSA (Weighted Finite-State Acceptor), FST, and WFST. These four categories can be collectively referred to as FSTs. Figure 1As shown, schematic diagrams of the four different types of FST are presented. In each FST, the bold black circle represents the initial state, the double black circle represents the final state, and the remaining circles represent normal states. The arrows in each FST represent an arc. It can be seen that each arc connects two states, and the direction of the arrow represents the timing of the transition from one state to another.

[0087] See Figure 1 In FSA, each arc has only an input label and no output label, and each arc has no weight, nor do the initial and final states. In FST, each arc has both input and output labels, but each arc has no weight, nor do the initial and final states. In WFSA, each arc has only an input label and no output label, but each arc has weight, as do the initial and final states. In WFST, each arc has both input and output labels, and each arc has weight, as do the initial and final states.

[0088] In any type of Free-Speed ​​Search (FST), a continuous transition path from any initial state to any final state is called a valid path in the FST. During the continuous state transitions along a valid path, if each arc of the valid path contains an output label, then this valid path will output a sequence of output labels. If each arc of the valid path also carries weights, then the weights will be continuously accumulated. For example, in... Figure 1 In WFST, for a valid path indicated by the input label sequence bcde, an output label sequence yxwv will be obtained, with a cumulative weight of 3.4 (the sum of the weights of the initial and final states and the weights on each arc of the valid path).

[0089] 2. Forward fractions of states in FST

[0090] For any weighted FST that does not form self-loops (i.e., the starting state and the arriving state are equal, that is, the two states connected by an arc are the same state), and has only one starting state and one ending state, a forward score can be calculated for each state.

[0091] The forward score calculation rule is defined by the following recursive logic: a) The forward score of the initial state is defined as the weight carried by the initial state itself; b) For other states, if there are n (n≥1) arcs leading to that state, then the forward score of that state is expressed as: Among them, s i w represents the forward fraction representing the starting state of the i-th (1≤i≤n) arc. i This represents the weight carried on the i-th arc. and These are two defined calculation rules, collectively referred to as the semining operation. Illustratively, in the case of using the log-semining operation, This refers to numerical addition. This refers to the logsumexp operation. Specifically, the logsumexp operation means that for m parameters {x1, x2, ..., x...}, ..., logsumexp... m}, calculate the logarithm of the sum of the exponents of each parameter: log(exp(x1)+...+exp(x...) m ), where m is an integer greater than or equal to 1, substitute each parameter into Perform logsumexp operations and recursively calculate the forward score for each state.

[0092] 3. FST form of lattice

[0093] A word lattice is a specific example of FST applied in the field of ASR. When an ASR neural network decodes acoustic features, its predictions for the recognized text and their associated scores can be compiled into a word lattice in the form of FST. The word lattice is used to provide character sequences of all possible candidate texts, which can be the same or different in length.

[0094] Schematic, each word lattice (predicted word lattice, word lattice to be operated on, and target word lattice, wherein the word lattice to be operated on includes a numerator word lattice and a denominator word lattice, and the target word lattice includes a target numerator word lattice and a target denominator word lattice) involved in the embodiments of this application can be provided as a WFSA, that is, each state in a word lattice can be regarded as a state of WFSA, each arc connecting two states in a word lattice can be regarded as an arc connecting two states in WFSA, each character sequence in a word lattice can be regarded as a legal path of WFSA, and each arc in the word lattice contains characters identified in the character sequence.

[0095] For the target word grid obtained after the intersection operation, the arcs at different vertical positions in the target word grid represent characters identified from different audio frames, and multiple arcs at the same vertical position represent multiple possible candidate characters identified from the same audio frame. In addition, the arcs starting from the initial state and the arcs reaching the final state in the target word grid represent the first and last characters in a candidate text, respectively.

[0096] The word lattice of the WFSA form described above can be completely represented by a set of all arcs in the lattice. Each arc in the set indicates the contextual relationship between a starting state and a destination state, where the starting state corresponds to the starting point of the arrow in the WFSA, and the destination state corresponds to the ending point of the arrow in the WFSA. Thus, each arc in the set can be uniquely represented by a quadruple {starting state index, destination state index, input label, weight}, where the input label can be provided as the receiving symbol index. By using a set of arcs to represent WFSA, it is possible to avoid explicitly storing each state in the WFSA when representing a word lattice of the WFSA form, but instead record the state (i.e., the predicted character) through the quadruple of each arc.

[0097] 4. Intersection operation between FSTs

[0098] For WFSA, the intersection operation is a common operation: it takes two WFSAs as input and outputs a new WFSA. The definition of the intersection operation is as follows: if any input label sequence can be received by both WFSAs simultaneously, then the new WFSA also accepts this input label sequence, and the cumulative weight of this input label sequence in the new WFSA is the result of calculating the cumulative weights of the input label sequence on the two input WFSAs. Here, the calculation result refers to the value corresponding to the weight accumulation rule in the segmentation operation.

[0099] like Figure 2 As shown, suppose we perform an intersection operation on a_fsa and b_fsa to obtain a new c_fsa, where a_fsa, b_fsa, and c_fsa are all WSFAs. It can be seen that for a_fsa and b_fsa, there is one and only one input label sequence {1, -1} that can be accepted by both a_fsa and b_fsa simultaneously. Therefore, c_fsa contains only this one valid path, and the weight of each arc in this valid path is the cumulative weight of the two arcs with the same input label on a_fsa and b_fsa. For example, the cumulative weight of the arc with input label 1 on c_fsa is 1.2, which is equal to the sum of its weight 0.2 on a_fsa and its weight 1 on b_fsa.

[0100] 5. LF-MMI based on FST approximation

[0101] The LF-MMI criterion is a common training criterion for ASR systems, but this application proposes an LF-MMI method that can also be applied to the decoding stage of ASR systems. The LF-MMI criterion expresses the principle that, given known acoustic features... At that time, this acoustic feature The recognized text is The posterior probability can be expressed by the following formula:

[0102]

[0103] in, This represents a known text of length U (typically, it's the labeled reference text during training and the predicted candidate text during decoding), meaning the text contains U characters. This represents an acoustic feature of length T (T≥1) frames, meaning the audio to be identified contains T audio frames. Let U represent an arbitrary text sequence of length U', and P(a|b) represent the conditional probability of event a occurring given that event b has occurred. a can be represented by acoustic features, and b can be represented by a character sequence.

[0104] In the LF-MMI criterion, in order to calculate the molecule in the above formula... and denominator The FST method can be used for approximation, that is, the molecule is based on text. Compile into a molecular WFSA, i.e., molecular clade. The denominator WFSA is a denominator lattice G that is consistent for any text. den Meanwhile, the output of the neural network (i.e., the language model) is also compiled into a WFSA, denoted as the prediction lattice nnet_output.

[0105] According to the FST method, the predicted word lattice nnet_output and the molecular word lattice are respectively... and denominator case G den Performing the intersection operation will yield two new WFSAs, referred to as the target numerator lattice and the target denominator lattice, respectively. The forward fractions of the final state are calculated on both the target numerator and target denominator lattices. The desired approximation posterior probability can be obtained by calculating the difference between the two lattices in the logarithmic domain or their quotient in the non-logarithmic domain.

[0106] The aforementioned LF-MMI criterion can be used not only in the training phase of an ASR system but also in the decoding phase. When using LF-MMI for decoding, for a given hypothesis... (That is, given candidate text, u represents a local hypothesis of a text of length U that is being decoded, but text U has not yet been fully decoded). Typically, the above-mentioned posterior probability approximation method based on FST is used to calculate the sequence of numbers related to t. t represents the audio frame number in the audio to be identified. Solving for the sequence of numbers related to t actually involves solving the numerator sequence of numbers related to t in the aforementioned formula. and the sequence of denominators with respect to t Clearly, since there are T possible values ​​for t, both the numerator and denominator sequences need to undergo T intersection operations with the prediction word lattice, totaling 2T intersection operations. However, intersection operations are computationally expensive, and the decoding phase requires repeatedly exploring various hypotheses. (That is, a posterior probability is calculated for each different candidate text), therefore, the computational cost of decoding based on the LF-MMI criterion is very high.

[0107] In view of this, the audio processing method provided in this application, when decoding the acoustic features of the audio to be identified based on the LF-MMI criterion, only needs to perform one global intersection operation on the numerator and denominator word lattices respectively using the predicted word lattice for a total of T frames of acoustic features in the aforementioned formula, and then recover the recognition result after the intersection operation frame by frame, instead of performing T intersection operations frame by frame. This can achieve the effect of saving computing resources and speeding up the decoding speed without losing decoding accuracy, and the decoding speed can be improved by several times.

[0108] The above-mentioned audio processing method has a wide range of applicable products and application scenarios. Any product involving ASR technology, such as speech recognition engines and related downstream products that integrate speech recognition engines, can use this decoding method to improve the decoding speed of the input audio to be recognized.

[0109] The system architecture of the embodiments of this application will be described below.

[0110] Figure 3 This is a schematic diagram illustrating the implementation environment of an audio processing method provided in an embodiment of this application. See also... Figure 3 The implementation environment includes a terminal 301 and a server 302, which can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this.

[0111] The terminal 301 has applications that support audio recognition services installed and running. For example, the application types include, but are not limited to: audio applications, audio and video applications, social applications, game applications, intelligent customer service, intelligent voice assistants, intelligent assistants, smart voice products, ASR products, browser applications, or other applications that integrate ASR function modules. This application embodiment does not specifically limit this.

[0112] In some embodiments, terminal 301 is a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0113] Server 302 provides backend services to applications that support audio recognition services. Optionally, server 301 includes at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, server 302 undertakes the primary ASR calculation work, and terminal 301 undertakes the secondary ASR calculation work; or, server 302 undertakes the secondary ASR calculation work, and terminal 301 undertakes the primary ASR calculation work; or, server 302 and terminal 301 use a distributed computing architecture to perform collaborative ASR calculation work.

[0114] In some embodiments, server 302 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0115] In an exemplary scenario, a user inputs audio to be recognized on terminal 301, triggering terminal 301 to send an audio recognition request carrying the audio to be recognized to server 302. In response to the audio recognition request, server 302 predicts the recognition text that matches the semantics of the audio to be recognized based on the audio processing method provided in this application embodiment. For example, it predicts the recognition character corresponding to each phoneme in the audio to be recognized, combines the recognition characters into recognition text, and returns an audio recognition response carrying the recognition text to terminal 301. This can save the local computing resources of terminal 301.

[0116] In another exemplary scenario, a user inputs audio to be recognized on terminal 301, triggering terminal 301 to perform automatic audio recognition locally using the ASR system. Terminal 301 predicts the recognition text that matches the semantics of the audio to be recognized based on the audio processing method provided in this application embodiment. For example, it predicts the recognition character corresponding to each phoneme in the audio to be recognized and combines the recognition characters into recognition text. This eliminates the need to connect to server 302 to access the audio recognition service and saves communication overhead.

[0117] The basic flow of the audio processing method according to the embodiments of this application will be described below.

[0118] Figure 4 This is a flowchart of an audio processing method provided in an embodiment of this application. See also... Figure 4 This embodiment is executed by an electronic device, which may be provided as terminal 301 or server 302 in the above-described implementation environment. This embodiment includes the following steps:

[0119] 401. An electronic device decodes multiple audio frames in the audio to be identified to obtain a predicted word grid for the multiple audio frames, which provides a character sequence of multiple candidate texts obtained from the identification of the multiple audio frames.

[0120] In some embodiments, the electronic device acquires the audio to be recognized. The audio to be recognized may be user input, stored in a local database, or downloaded from a cloud database. This application embodiment does not specifically limit the source of the audio to be recognized. In one example, a user launches an application that supports audio recognition services. After the user has fully authorized and consented to microphone permissions, the user can start recording by clicking the recording control, and then reads a passage of audio to be recognized. The electronic device then captures this audio through the microphone.

[0121] In some embodiments, the electronic device preprocesses the audio to be identified to obtain multiple audio frames in the audio to be identified. Optionally, the preprocessing operations include, but are not limited to, VAD (Voice Activity Detection), pre-emphasis, framing, frame shifting, windowing, Fourier transform, etc. The specific flow of the preprocessing operations is not limited here.

[0122] In some embodiments, an ASR system is configured in the application of the electronic device. The ASR system involves a trained ASR neural network. Multiple audio frames are input into the ASR neural network, and the acoustic features of the multiple audio frames are extracted. Then, based on the acoustic features of the multiple audio frames, the output distribution of the ASR neural network is compiled into a prediction word grid in FST form. The prediction word grid is used to provide character sequences of multiple candidate texts predicted by the language model. These character sequences can be of the same or different lengths. Each character sequence includes one or more characters. The vertical position of each character in the prediction word grid is used to identify the position of the audio frame to which the character belongs. For example, the arc in the first column of the prediction word grid corresponds to the recognized character of the first audio frame, the arc in the second column of the prediction word grid corresponds to the recognized character of the second audio frame, and so on. The ASR neural network can be trained locally on the electronic device or trained on a server and then distributed to the electronic device. During training, ML criteria, MMI criteria, or LF-MMI criteria can be used, etc., and this application embodiment does not specifically limit this.

[0123] 402. The electronic device obtains a word cell to be operated on based on any candidate text of the plurality of audio frames. The word cell to be operated on is used to provide the character sequence of the candidate text or the global character sequence of different candidate texts.

[0124] In the embodiments of this application, the word lattice to be operated refers to the numerator word lattice compiled from any candidate text and a global denominator word lattice. Since the processing flow of the numerator word lattice and the denominator word lattice is the same, the numerator word lattice and the denominator word lattice are collectively referred to as the word lattice to be operated, which can be regarded as a WFSA to be operated in essence.

[0125] In some embodiments, for any predicted candidate text, the electronic device can compile a numerator lattice and a denominator lattice, such that the denominator lattice remains consistent for any candidate text, but the numerator lattice changes with different candidate texts.

[0126] 403. The electronic device performs an intersection operation on the predicted word grid and the word grid to be operated on to obtain the target word grid and path relationship information. The path relationship information indicates the association relationship between each arc in the target word grid and the same arc in the word grid to be operated on. Each arc in the target word grid indicates a character.

[0127] In this context, each arc in the word lattice is used to connect the contextual relationship between a starting state and a destination state. For word lattices in WFSA form, the starting state is equivalent to the starting point of the arrow in the WFSA, and the destination state is equivalent to the ending point of the arrow in the WFSA.

[0128] In some embodiments, the electronic device performs only one intersection operation on the predicted word grid and the word grid to be operated on, generating a new target word grid. This target word grid contains only the input label sequence that is accepted by both the predicted and target word grids; in other words, it contains only the legal paths common to both. Simultaneously, during this intersection operation, an additional byproduct, path relationship information, is generated. This information indicates the association between each arc in the target word grid and the same arc in the word grid to be operated on. That is, although the arcs in the target word grid are necessarily included in both the predicted and target word grids, the index of the same arc may change in different word grids. Therefore, during the intersection operation, the correspondence of the same arc in the target and target word grids is also recorded, representing which arcs are related in the target and target word grids before and after the intersection operation, thus indicating the correspondence of arcs in the two word grids and obtaining the aforementioned path relationship information.

[0129] 404. The electronic device determines the prediction scores of multiple states in the target word lattice based on the target word lattice and the path relationship information. The prediction scores indicate the degree of semantic matching between each state and the preceding audio frame in the audio to be identified.

[0130] In some embodiments, the electronic device can calculate the prediction score between each state in the target word grid and the preceding audio frame relative to its own position using the target word grid and path relationship information obtained after performing only one intersection operation. This prediction score indicates the degree of semantic matching between the state and the preceding audio frame at the corresponding position. For example, for any state in the i-th column (1≤i≤T) of the target word grid, the prediction score of this state indicates the degree of semantic matching between the state and the previous i audio frames (i.e., the first frame to the i-th frame), representing the posterior probability that the previous i frames reach the i-th state through a legitimate path. This is equivalent to assigning and distributing the prediction score at the state level for the entire target word grid, which facilitates the recovery of the recognition result of each audio frame frame by frame. T is the number of audio frames contained in the audio to be recognized, and T is a positive integer greater than or equal to 1.

[0131] 405. The electronic device determines the prediction probability of the candidate text based on the prediction scores of multiple states in the target word lattice. The prediction probability characterizes the degree of semantic matching between the candidate text and the audio to be identified.

[0132] In some embodiments, since each candidate text is a character sequence consisting of all the characters traversed by a legal path, and each arc on the legal path has a corresponding character, the electronic device can determine the prediction probability of the entire legal path based on the prediction scores of each state in the target word grid. The prediction probability of each legal path represents the prediction probability of the candidate text consisting of the character sequence corresponding to that legal path. By comparing the prediction probabilities of multiple candidate texts horizontally, the candidate text with the highest prediction probability can be selected from multiple candidate texts, and the candidate text with the highest prediction probability is output as the recognition text of the audio to be recognized.

[0133] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0134] The technical solution provided in this application embodiment performs only one intersection operation on the predicted word grid and the word grid to be operated on. Using the target word grid obtained from the one intersection operation and the path relationship information as its byproduct, a prediction score can be assigned to each state in the target word grid. This allows the prediction probability of the candidate text to be easily reconstructed frame by frame using the prediction scores of each state. This enables the selection of the recognition text with the highest semantic matching degree based on the prediction probability. In this way, without sacrificing the audio decoding accuracy, the computational overhead is greatly compressed by one intersection operation, thereby greatly saving computing resources and improving the computing speed, thus improving the audio decoding speed.

[0135] The audio processing method of this application embodiment will be described in detail below with specific examples of the LF-MMI criterion.

[0136] Figure 5 This is a flowchart of an audio processing method provided in an embodiment of this application. See also... Figure 5 This embodiment is executed by an electronic device, which may be provided as terminal 301 or server 302 in the above-described implementation environment. This embodiment includes the following steps:

[0137] 501. The electronic device inputs multiple audio frames from the audio to be identified into an ASR neural network, and extracts the acoustic features of the multiple audio frames through the ASR neural network.

[0138] For details on how to obtain the audio to be identified and how to obtain the audio frames, please refer to the description in step 401. It will not be repeated here.

[0139] In some embodiments, the electronic device locally stores the parameter matrix of the ASR neural network, which is used to support the audio recognition function in the ASR system. The ASR system can be a functional module embedded in an application that supports audio recognition services, or it can be a functional module that can be independently called in the operating system.

[0140] In some embodiments, the ASR neural network can be applied by the electronic device after local training, or the parameter matrix of the ASR neural network is downloaded to the electronic device after training on a cloud server. This application embodiment does not specifically limit this. The ASR neural network can use ML criteria, MMI criteria, LF-MMI criteria, etc. during training, which are not limited here.

[0141] In some embodiments, the ASR system is also provided as an API (Application Programming Interface) that can be invoked. The main thread of the application calls the API of the ASR system, thereby sending an audio recognition request carrying the audio to be recognized to the cloud server. The cloud server then executes the subsequent audio recognition process and returns the recognized text to the electronic device after obtaining it.

[0142] In this embodiment of the application, only the ASR process implemented locally by the electronic device is used as an example for illustration. However, the ASR process can also be implemented through communication and interaction between the electronic device and the server. No specific limitation is made here.

[0143] In some embodiments, after preprocessing the audio to be recognized, the electronic device obtains multiple audio frames. These multiple audio frames are then input into an ASR neural network to extract acoustic features from the audio frames. For example, for T audio frames in the audio to be recognized, the T audio frames are input into the ASR neural network to obtain an acoustic feature sequence of length T. T represents the number of audio frames in the audio to be identified, and T is an integer greater than or equal to 1.

[0144] 502. The electronic device decodes the acoustic features through the ASR neural network to obtain the predicted word grid of the multiple audio frames. The predicted word grid provides the character sequence of multiple candidate texts identified from the multiple audio frames.

[0145] In some embodiments, the electronic device uses an ASR neural network to decode the acoustic features of multiple audio frames extracted in step 501 to predict a matrix consisting of a set of frame-by-frame probability distributions (i.e., the output distribution of the ASR neural network), and compiles it into a prediction lattice nnet_output. The prediction lattice nnet_output can be regarded as converting the neural network output distribution of the ASR neural network into a WFSA-form lattice. Based on the prediction lattice, the final recognized text of the audio to be recognized can be further filtered. Figure 6 The diagram shown illustrates a prediction lattice nnet_output. It should be noted that there may be hundreds of arcs between characters 0 and 1. Figure 6 For simplicity, only three arcs are drawn as an illustration.

[0146] In steps 501-502 above, a possible implementation method is provided to decode multiple audio frames in the audio to be recognized and obtain the predicted word lattice of the multiple audio frames. Through the ASR system in the form of a neural network, the output distribution of the ASR neural network can be easily compiled into a predicted word lattice in the form of WFSA, which facilitates the approximation of the posterior probability of the LF-MMI criterion based on the WFSA method, and realizes end-to-end audio recognition from the audio to be recognized to the recognized text.

[0147] In other embodiments, ASR systems with other architectures can also be used to extract predicted word lattices from multiple audio frames. For example, some ASR systems with explicit acoustic model-language model use the acoustic model to extract acoustic features and then use the language model to decode the recognized text. The predicted word lattices need to be calculated during the process of obtaining the recognized text. The embodiments of this application do not specifically limit the architecture of the ASR system.

[0148] 503. The electronic device obtains a word cell to be operated on based on any candidate text of the multiple audio frames. The word cell to be operated on is used to provide the character sequence of the candidate text or the global character sequence of different candidate texts.

[0149] In some embodiments, the term lattice to be operated on includes a numerator term lattice and a denominator term lattice according to the LF-MMI criterion. The numerator term lattice provides the character sequence of the candidate text, and the denominator term lattice provides the global character sequence of different candidate texts. In other words, when the term lattice to be operated on is used to provide the character sequence of the current candidate text, the term lattice to be operated on is provided as a numerator term lattice, and when the term lattice to be operated on is used to provide the global character sequence of different candidate texts, the term lattice to be operated on is provided as a denominator term lattice.

[0150] In some embodiments, for any candidate text predicted by the semantic model, a numerator lattice can be compiled based on the candidate text, and a denominator lattice can also be compiled, so that the denominator lattice remains consistent for any candidate text, but the numerator lattice changes with different candidate texts.

[0151] In some embodiments, based on any candidate text predicted by the semantic model, its sub-word lattice and denominator word lattice are obtained. Since the processing flow of intersection operation between the sub-word lattice and the denominator word lattice and the predicted word lattice is the same, the sub-word lattice and the denominator word lattice are collectively referred to as the word lattice to be operated on, a_fsa, for description. The word lattice to be operated on can be substituted into the sub-word lattice or the denominator word lattice; no specific limitation is made here. Figure 7 As shown, a schematic diagram of a lattice a_fsa to be operated on is presented for candidate text of length U. The lattice to be operated on, a_fsa, can be the molecular lattice in the LF-MMI criterion. It can also be the denominator clade G in the LF-MMI criterion. den .

[0152] 504. The electronic device performs an intersection operation on the predicted word grid and the word grid to be operated on to obtain the target word grid and path relationship information. The path relationship information indicates the association relationship between each arc in the target word grid and the same arc in the word grid to be operated on. Each arc in the target word grid indicates a character.

[0153] In some embodiments, since the word lattice to be operated on includes a numerator word lattice and a denominator word lattice, the electronic device can perform only one intersection operation with each of the predicted word lattice and the numerator word lattice. The operation method of the intersection operation is described in the foregoing embodiments and will not be repeated here.

[0154] In some embodiments, the electronic device performs only one intersection operation on the predicted word lattice and the molecular word lattice to obtain the target molecular word lattice and molecular path relationship information. The molecular path relationship information indicates the association between each arc in the target molecular word lattice and the same arc in the molecular word lattice.

[0155] In some embodiments, the electronic device performs only one intersection operation on the predicted word grid and the denominator word grid to obtain the target denominator word grid and denominator path relationship information. The denominator path relationship information indicates the association between each arc in the target denominator word grid and the same arc in the denominator word grid.

[0156] As can be seen, since the intersection operation between the predicted word lattice and the numerator and denominator word lattices is the same, for the sake of simplicity, the numerator and denominator word lattices are collectively referred to as the word lattice to be operated on, the target numerator and target denominator word lattices are collectively referred to as the target word lattice, and the numerator path relationship information and the denominator path relationship information are collectively referred to as the path relationship information Arc_map. The path relationship information Arc_map is used to indicate the correspondence between each arc on the target word lattice and the same arc on the word lattice to be operated on.

[0157] In one example, performing one intersection operation on the predicted word lattice `nnet_output` and the word lattice to be operated on `a_fsa` yields a target word lattice and a crucial byproduct: path relationship information `Arc_map`. For example, for... Figure 6 The predicted word lattice nnet_output and Figure 7 Performing one intersection operation on the lattice a_fsa shown above will yield a result like... Figure 8The target word lattice is shown, along with a byproduct—the path relationship information Arc_map on the target word lattice. For example, the path relationship information Arc_map = {0,1,2,2,5,6,7,8,9,7,13,15,16,19,17,22,27}. Since the path relationship information Arc_map represents the correspondence between each arc on the target word lattice and the same arc on the word lattice to be operated on, it is clear that the last arc on the target word lattice corresponds to the 27th arc on the word lattice to be operated on.

[0158] It should be noted that the aforementioned prediction lattice nnet_output, the lattice to be operated on a_fsa, and the target lattice are all WFSA-type lattices, consisting of several arcs. Each arc has an index in its own lattice, so that in the set of arcs used to store the lattices, the quadruple of any arc can be found in its own lattice by using the index of any arc. In other words, the index of an arc in the lattice is equivalent to the index of the quadruple of the arc in the set of arcs.

[0159] 505. The electronic device filters each arc in the target word grid to obtain multiple target arcs, so that each state in the target word grid has one and only one target arc that leads to that state.

[0160] In some embodiments, for any state on the target word lattice, there are often multiple arcs originating from different starting states that reach that state. Therefore, it is necessary to filter all arcs on the target word lattice to ensure that each state in the target word lattice has exactly one target arc that can reach it. Finally, the pruned target word lattice composed of all target arcs is selected. For example, after selecting all target arcs in the target word lattice, the remaining quadruples that are not target arcs are deleted from the arc set, retaining only all target arcs. In one example, such as... Figure 8 The target lattice shown exists as follows: although there is an arc from state 1, 2, and 3 to state 4, after pruning, only the arc from state 1 to state 4 is retained as the target arc. Similarly, only one such target arc is retained for all other states.

[0161] 506. Based on the path relationship information, the electronic device determines the associated arc of each target arc from the word cell to be operated on, wherein the target arc and the associated arc indicate the same character in the word cell.

[0162] In some embodiments, for each target arc obtained after pruning in step 505, the path relationship information Arc_map from step 504 can be used to look up the index of the associated arc in the word lattice to be operated on, based on the index of the target arc in the target word lattice. This is equivalent to finding all associated arcs corresponding to all target arcs in the word lattice to be operated on, through the path relationship information Arc_map. Each target arc has one and only one associated arc. The target arc and associated arc indicate the same character in the word lattice; that is, the starting state, arrival state, and input label of the target arc and associated arc are the same, but the weights on the target arc and associated arc can be different. In one example, such as... Figure 8 The target lattice shown is pruned, retaining only the arcs from state 1 to state 4 as the target arcs. Looking up the path relationship information Arc_map = {0,1,2,2,5,6,7,8,9,7,13,15,16,19,17,22,27}, we find that the value of the 4th arc in the Arc_map is 2, indicating that the associated arc of the target arc is the 2nd arc on the lattice to be operated on, i.e., as shown... Figure 7 The arc from state 0 to state 2 in the lattice a_fsa shown is the aforementioned associated arc.

[0163] 507. The electronic device assigns a predicted score to the target word lattice based on the forward score of the arrival state connected by each associated arc in the word lattice to be operated.

[0164] In some embodiments, the electronic device can acquire the arrival state of each associated arc in WFSA, calculate the forward score of the arrival state on the lattice to be operated on, and use the forward score of the arrival state on the lattice to be operated on to assign a prediction score of the arrival state on the target lattice.

[0165] The following steps, A1 to A2, will explain how to calculate the forward fraction of the arrival state in any associated arc on the lattice a_fsa to be operated on.

[0166] A1. The electronic device determines the arrival status of each associated arc in the word cell to be operated.

[0167] In some embodiments, for each associated arc on the word cell a_fsa found in step 506, the arrival state of this associated arc is found on the word cell a_fsa, that is, the arrival state of this associated arc is found on the word cell a_fsa (i.e., the end point of the arc's arrow represents the arrival state, and conversely, the start point of the arc's arrow represents the departure state). In one example, such as Figure 7In the lattice of the word to be operated on, the arc from state 0 to state 2 is an associated arc. Therefore, state 2 is determined to be the arrival state connected by the associated arc. Obviously, there is also a similar arrival state (i.e., state 4) in the target lattice. Although the state number has changed, state 2 is the arrival state of the associated arc, and state 4 is the arrival state of the target arc. Therefore, state 4 in the target lattice and state 2 in the lattice of the word to be operated on are the same arrival state.

[0168] A2. The electronic device obtains the forward score of the arrival state based on each arc in the word lattice that leads to the arrival state.

[0169] In some embodiments, the electronic device performs a recursive forward calculation of the forward fraction of the arrival state based on all arcs on the lattice a_fsa to the arrival state, in order to obtain a forward fraction of the arrival state.

[0170] The following section will explain how to calculate the forward fraction for any reached state, using sub-steps A21 to A23 as examples.

[0171] A21. For each arc in the word lattice that has reached the arrival state, the electronic device adds the forward fraction of the starting state connected to each arc and the weight on each arc to obtain the forward weight parameter of each arc.

[0172] In some embodiments, assuming there are n (n≥1) arcs in the lattice a_fsa to reach the reaching state, then for each of the n arcs, the forward fraction of the starting state (i.e., the starting point of the arc's arrow) connected to this arc is added to the weight on this arc to obtain the forward weight parameter of this arc. For example, for the i-th (1≤i≤n) arc among the n arcs, the forward fraction s of the starting state connected to the i-th arc is... i The weight w on the i-th arc i Add them together to obtain the forward weight parameters of the i-th arc. This refers to numerical addition. It should be noted that the forward fraction s of the starting state... i The calculation is also performed in the same way through sub-steps A21 to A23, so that the forward fraction of the starting state can be calculated recursively until the initial state is calculated, at which point the forward fraction of the initial state is set to the weight carried by the initial state itself.

[0173] A22. Electronic devices use the natural base as the base and the forward weight parameter of each arc as the exponent to obtain the forward weight index of each arc.

[0174] In some embodiments, the forward weight index of each arc is calculated using the natural base e as the base and the forward weight parameter calculated for each arc in sub-step A21 as the exponent. For example, for the i-th (1≤i≤n) arc among the above n arcs, the forward weight index of the i-th arc is calculated using the natural base e as the base and the forward weight parameter of the i-th arc as the exponent. For the index, obtain the forward weight index of the i-th arc.

[0175] A23. The electronic device takes the logarithm of the sum of the forward weight indices of each arc to obtain the forward score of the reached state.

[0176] In some embodiments, for each of the n arcs that reach the arrival state, a forward weight index is calculated through steps A21-A22. The n forward weight indices of these n arcs are summed to obtain a sum value. Taking the logarithm of this sum value yields the forward score of the arrival state. In one example, the n forward weight indices of the aforementioned n arcs are summed to obtain a sum value. Taking the logarithm of the sum gives the forward fraction of the reached state.

[0177] For simplicity, the forward fraction of the reached state is denoted as in, This refers to numerical addition. This refers to the logsumexp operation.

[0178] In the above sub-steps A21 to A23, a possible implementation method is provided for recursively calculating the forward fraction of each reached state. and There are two defined calculation rules, collectively referred to as the semiring operation. Here, we will only take the case of using log-semiring operation as an example for explanation, but other semiring operations can also be used. This application does not specifically limit this.

[0179] In steps A1 to A2 above, a possible implementation is provided on how to calculate the forward fraction of the arrival state in the word lattice a_fsa based on all arcs that have reached the arrival state, according to the arrival state of the associated arc. Optionally, it is also possible not to use all arcs that have reached the arrival state, but to perform certain pruning or compression, which is not specifically limited here.

[0180] The following section, in conjunction with steps B1 to B3, explains how to use the forward score of each arrival state in the word lattice to be operated on to assign the prediction score of the same arrival state in the target word lattice.

[0181] B1. The electronic device queries each state of the word frame to be operated on and obtains multiple candidate states that meet the query conditions. The query conditions indicate that there is an arc that starts from the current state and reaches the final state. The arc that reaches the final state indicates the character recognized in the last audio frame among the multiple audio frames.

[0182] In some embodiments, the electronic device can filter all states in the word array a_fsa to determine all candidate states in a_fsa that satisfy the following query conditions: there exists an arc starting from the current state, and its destination state is the final state. The final state refers to the state located in the last column of the word array a_fsa. Each arc reaching the final state represents a possible character identified from the last audio frame, and the candidate states are equivalent to the starting states of the arcs reaching the final state. Traversing all the candidate states queried in a_fsa allows all candidate states to form a dictionary, where the key of the dictionary is the character index, and the value is the weight of the arc starting from the current state and reaching the final state. It should be noted that for each candidate state in a_fsa that meets the query conditions, there is one and only one arc starting from the current state and reaching the final state. Figure 7 Taking the lattice a_fsa as an example, it can be seen that states 5, 7, 9, and 10 in the lattice a_fsa are all candidate states that meet the query conditions. Using the state numbers 5, 7, 9, and 10 as keys and the weights of each candidate state reaching the final state (-1.4, -1.4, -0.05, -0.05) as values, a dictionary containing 4 candidate states can be constructed.

[0183] B2. When the arrival state belongs to the candidate state, the electronic device adds the weight of the arc from the arrival state to the terminal character to the forward score of the arrival state to obtain the predicted score of the arrival state.

[0184] In some embodiments, the electronic device uses the character sequence number of the arrival state as the key to search the dictionary. If a value corresponding to the key is found, it means that the arrival state belongs to the candidate state recorded in the dictionary. Then, the weight of the candidate state on the arc to the terminal state recorded in the value is read, and the weight read from the value is summed with the forward score of the arrival state calculated by steps A1 to A2. The sum is assigned to the prediction score of the same arrival state in the target lattice.

[0185] B3. If the electronic device does not belong to the candidate state when the arrival state, it will assign a preset value to the predicted score of the arrival state.

[0186] In some embodiments, the electronic device uses the character sequence number of the arrival state as the key to search the dictionary. If no value corresponding to the key is found, it means that the arrival state does not belong to the candidate states recorded in the dictionary. In this case, a preset value can be directly assigned to the prediction score of the same arrival state in the target lattice. For example, the preset value can be negative infinity or other minimum values. This application does not specifically limit the setting method of the preset value.

[0187] In steps B1 to B3 above, for each state in the target lattice, a pruned target arc can be found. Through the correspondence between the target arc and the associated arc indicated by the path relationship information Arc_map, a corresponding associated arc is found in the lattice to be operated on, a_fsa. Thus, the state reached by this associated arc is the same state in the lattice to be operated on (usually the state numbers of the two states are different). Then, using the forward score of the reached state in the lattice to be operated on, a_fsa, and the query result of the reached state in the dictionary to be looked up, it is determined whether to add the forward score to the weight recorded in Value in the dictionary to be looked up and assign it to the predicted score of the same state, or to directly modify the predicted score of the same state to negative infinity. This achieves accurate assignment of the predicted score of each state in the target lattice.

[0188] In one example, state 4 in the target lattice corresponds to state 2 in the lattice to be operated on, a_fsa. Since state 2 cannot directly reach the final state in the lattice to be operated on, state 2 is not a candidate state in the dictionary to be consulted. Therefore, the predicted score of state 4 in the target lattice is directly modified to negative infinity. Assuming that state 4 in the target lattice corresponds to any one of states 5, 7, 9, or 10 in the lattice to be operated on, then the forward score of the corresponding state should be added to the weight recorded in Value in the dictionary to be consulted to obtain the predicted score of state 4 in the target lattice.

[0189] In steps 505-507 above, a possible implementation is provided for determining the prediction scores of multiple states in the target word lattice based on the target word lattice and the path relationship information, wherein the prediction score indicates the degree of semantic matching between each state and the preceding audio frame in the audio to be identified. Although only one intersection operation is performed between the word lattice to be operated on and the prediction word lattice, the assignment of prediction scores to all states in the target word lattice becomes easy to implement due to the full construction and utilization of the byproduct, namely the path relationship information. This greatly benefits the reconstruction of the prediction probability of the entire candidate text.

[0190] 508. The electronic device determines the prediction probability of the candidate text based on the prediction scores of multiple states in the target word lattice. The prediction probability characterizes the degree of semantic matching between the candidate text and the audio to be identified.

[0191] In some embodiments, since the word lattice to be operated on actually includes a numerator word lattice and a denominator word lattice, and the target word lattice also includes a target numerator word lattice and a target denominator word lattice, the reconstruction process of the prediction probability of the entire candidate text will be described below in conjunction with steps C1 to C4.

[0192] C1. For each of the multiple audio frames, the electronic device obtains the molecular prediction component of the audio frame based on the target molecular lattice.

[0193] In some embodiments, for each state in the target molecular lexicon, a prediction score can be calculated in steps 505-507. Since several states in the target molecular lexicon at the same vertical position belong to the same audio frame, the molecular prediction component of the audio frame can be obtained by using the logsumexp method to calculate the prediction score for all states in the target molecular lexicon that belong to any audio frame.

[0194] In some embodiments, using the logsumexp method to calculate the prediction score means: for each of the plurality of audio frames, query at least one candidate molecular state identified for that audio frame from the target molecular lexicon; then, using the natural base as the base and the prediction score of each candidate molecular state as the exponent, obtain the prediction index of each candidate molecular state; then, take the logarithm of the sum of the prediction indices of the at least one candidate molecular state to obtain the molecular prediction component of the audio frame. The logsumexp method is the same as the method described in steps A22 to A23, and will not be repeated here.

[0195] Because the logsumexp method can fully fuse the prediction scores of different candidate molecular states in the same audio frame, the accuracy of the molecular prediction components in each audio frame is guaranteed.

[0196] C2. The electronic device obtains the denominator prediction component of the audio frame based on the target denominator lattice.

[0197] In some embodiments, for each state in the target denominator lattice, a prediction score can be calculated in steps 505-507. Since several states in the target denominator lattice that are at the same vertical position belong to the same audio frame, the denominator prediction component of this audio frame can be obtained by using the logsumexp method to calculate the prediction score for all states in the target denominator lattice that belong to any audio frame.

[0198] In some embodiments, using the logsumexp method to calculate the predicted score means: for each of the plurality of audio frames, query at least one candidate denominator state identified for that audio frame from the target denominator lattice; then, using the natural base as the base and the predicted score of each candidate denominator state as the exponent, obtain the predicted exponent of each candidate denominator state; then, take the logarithm of the sum of the predicted exponents of the at least one candidate denominator state to obtain the denominator predicted component of that audio frame. The logsumexp method is similar to the method described in steps A22 to A23, and will not be repeated here.

[0199] Because the logsumexp method can fully integrate the predicted scores of different candidate denominator states of the same audio frame, the accuracy of the denominator prediction component of each audio frame is guaranteed.

[0200] C3. The electronic device determines the single-frame prediction component of the audio frame based on the numerator and denominator prediction components of the audio frame.

[0201] In some embodiments, for each of the plurality of audio frames, the single-frame prediction component of the audio frame is obtained by subtracting the denominator prediction component calculated in step C2 from the numerator prediction component calculated in step C1. Taking the t-th audio frame as an example, the single-frame prediction component of the t-th audio frame represents the posterior probability of the LF-MMI criterion. The prediction score at time t, where T represents the number of audio frames of the audio to be identified.

[0202] C4. The electronic device obtains the prediction probability of the candidate text based on the single-frame prediction components of each of the multiple audio frames.

[0203] In some embodiments, the single-frame predicted component is calculated for each of the T audio frames using steps C1 to C3 described above. Thus, even though only one intersection operation is performed between the predicted word lattice and each of the numerator and denominator word lattices to obtain a target numerator and a target denominator word lattice, the single-frame predicted component for any audio frame at any time t in the entire candidate text can be reconstructed using path relationship information. The overall process does not require T intersection operations with each of the numerator and denominator word lattices, greatly saving computational overhead and improving computational efficiency.

[0204] Furthermore, by utilizing the individual frame prediction components of the T audio frames, the prediction probability of the entire candidate text can be obtained. For example, the prediction probability of the entire candidate text can be obtained by performing an arithmetic average, weighted average, or weighted summation on the individual frame prediction components of the T audio frames. No specific limitations are imposed on this here.

[0205] 509. The electronic device filters multiple candidate texts based on their respective predicted probabilities to obtain the recognized text for the audio output to be recognized.

[0206] In some embodiments, the electronic device can calculate the predicted probability of each candidate text based on steps 501 to 508. By repeatedly performing the above operation, the predicted probabilities of some or all candidate texts can be calculated. Based on the predicted probabilities of each candidate text, these candidate texts can be filtered, such as selecting the candidate text with the highest predicted probability as the recognition text for the audio output to be recognized, thereby ensuring the highest degree of semantic matching between the recognition text and the audio to be recognized.

[0207] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0208] The technical solution provided in this application embodiment performs only one intersection operation on the predicted word grid and the word grid to be operated on. Using the target word grid obtained from the one intersection operation and the path relationship information as its byproduct, a prediction score can be assigned to each state in the target word grid. This allows the prediction probability of the candidate text to be easily reconstructed frame by frame using the prediction scores of each state. This enables the selection of the recognition text with the highest semantic matching degree based on the prediction probability. In this way, without sacrificing the audio decoding accuracy, the computational overhead is greatly compressed by one intersection operation, thereby greatly saving computing resources and improving the computing speed, thus improving the audio decoding speed.

[0209] During testing, within the AED (Attention-based Encoder-Decoder) framework, joint decoding was performed using the MMI Prefix Score (maximizing mutual information forward score) based on the LF-MMI criterion. This involved calculating the aforementioned numerator sequence with respect to t. and the sequence of denominators with respect to t The numerator sequence with respect to t Approximating the denominator sequence with respect to t using the numerator prediction component of T-frames. Using T-frame denominator prediction component approximation, the single-frame prediction component of each audio frame can be reconstructed by performing only two intersection operations (one for the numerator and one for the denominator), thus representing the posterior probability of the LF-MMI criterion. The predicted score at time t.

[0210] The technical solution of this application embodiment was tested on the common Chinese speech recognition dataset Aishell, and the experimental results are shown in Table 1 below.

[0211] Table 1

[0212] Before using this solution 4.4 4.8 116.9 After using this solution 4.4 4.8 7.47

[0213] Among them, the average real-time factor (RTF) refers to the Real Time Factor. The lower the RTF value, the better the real-time performance and the faster the decoding speed. The CER refers to the Character Error Rate. The lower the CER value, the higher the decoding accuracy.

[0214] It can be seen that the accuracy of CER indication in speech recognition did not change before and after using this solution, but the average real-time rate of the ASR system dropped significantly from 116.9 to 7.47, a difference of more than 15 times. Therefore, the decoding efficiency of ASR was significantly improved, that is, the audio recognition speed of ASR was greatly improved.

[0215] Figure 9 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application. Please refer to it. Figure 9 The device includes:

[0216] The decoding module 901 is used to decode multiple audio frames in the audio to be identified, and obtain the predicted word grid of the multiple audio frames. The predicted word grid provides the character sequence of multiple candidate texts obtained from the identification of the multiple audio frames.

[0217] The acquisition module 902 is used to acquire a word cell to be operated on based on any candidate text of the multiple audio frames. The word cell to be operated on is used to provide the character sequence of the candidate text or the global character sequence of different candidate texts.

[0218] Intersection operation module 903 is used to perform an intersection operation on the predicted word grid and the word grid to be operated on to obtain the target word grid and path relationship information. The path relationship information indicates the association relationship between each arc in the target word grid and the same arc in the word grid to be operated on. Each arc in the target word grid indicates a character.

[0219] The score determination module 904 is used to determine the predicted scores of multiple states in the target word grid based on the target word grid and the path relationship information. The predicted scores indicate the degree of semantic matching between each state and the preceding audio frame in the audio to be identified.

[0220] The probability determination module 905 is used to determine the prediction probability of the candidate text based on the prediction scores of multiple states in the target word lattice. The prediction probability characterizes the degree of semantic matching between the candidate text and the audio to be identified.

[0221] The technical solution provided in this application embodiment performs only one intersection operation on the predicted word grid and the word grid to be operated on. Using the target word grid obtained from the one intersection operation and the path relationship information as its byproduct, a prediction score can be assigned to each state in the target word grid. This allows the prediction probability of the candidate text to be easily reconstructed frame by frame using the prediction scores of each state. This enables the selection of the recognition text with the highest semantic matching degree based on the prediction probability. In this way, without sacrificing the audio decoding accuracy, the computational overhead is greatly compressed by one intersection operation, thereby greatly saving computing resources and improving the computing speed, thus improving the audio decoding speed.

[0222] In some embodiments, based on Figure 9 The device comprises the following components: the score determination module 904 includes:

[0223] The filtering unit is used to filter each arc in the target word grid to obtain multiple target arcs, so that each state in the target word grid has one and only one target arc that reaches that state;

[0224] An arc determination unit is used to determine, based on the path relationship information, the associated arc of each target arc from the word grid to be operated on, wherein the target arc and the associated arc indicate the same character in the word grid;

[0225] The assignment unit is used to assign a value to the predicted score of the arrival state in the target word lattice based on the forward score of the arrival state connected by each associated arc in the word lattice to be operated.

[0226] In some embodiments, based on Figure 9 The device comprises, wherein the score determination module 904 further includes:

[0227] The state determination unit is used to determine the arrival state connected to each associated arc in the word lattice to be operated on;

[0228] The score acquisition unit is used to acquire the forward score of the arrival state based on each arc in the word lattice that has reached the arrival state.

[0229] In some embodiments, the score acquisition unit is used for:

[0230] For each arc in the word lattice that reaches the arrival state, add the forward fraction of the starting state connected to each arc and the weight on each arc to obtain the forward weight parameter of each arc;

[0231] Using the natural base as the base and the forward weight parameter of each arc as the exponent, obtain the forward weight index of each arc;

[0232] Take the logarithm of the sum of the forward weight indices of each arc to obtain the forward score of the following character.

[0233] In some embodiments, the assignment unit is used for:

[0234] The query is performed on each state of the word to be operated on, and multiple candidate states that meet the query conditions are obtained. The query conditions indicate that there is an arc that starts from the current state and reaches the final state. The arc that reaches the final state indicates the character identified in the last audio frame among the multiple audio frames.

[0235] If the arrival state belongs to the candidate state, the weight on the arc from the arrival state to the terminal character is added to the forward score of the arrival state to obtain the predicted score of the arrival state.

[0236] If the arrival state does not belong to the candidate state, a preset value is assigned to the predicted score of the arrival state.

[0237] In some embodiments, the word grid to be operated on includes a numerator word grid and a denominator word grid, wherein the numerator word grid provides the character sequence of the candidate text and the denominator word grid provides the global character sequence of different candidate texts;

[0238] The intersection operation module 903 is used for:

[0239] Perform an intersection operation between the predicted word lattice and the molecular word lattice to obtain the target molecular word lattice.

[0240] Perform an intersection operation on the predicted word lattice and the denominator word lattice to obtain the target denominator word lattice.

[0241] In some embodiments, based on Figure 9 The device comprises a probability determination module 905 including:

[0242] The first acquisition unit is used to acquire the molecular prediction component of each audio frame in the plurality of audio frames based on the target molecular word lattice.

[0243] The second acquisition unit is used to acquire the denominator prediction component of the audio frame based on the target denominator word lattice.

[0244] The prediction component determination unit is used to determine the single-frame prediction component of the audio frame based on the numerator prediction component and the denominator prediction component of the audio frame.

[0245] The probability acquisition unit is used to obtain the prediction probability of the candidate text based on the single-frame prediction components of each of the multiple audio frames.

[0246] In some embodiments, the first acquisition unit is used for:

[0247] From the target molecule lattice, query at least one candidate molecule state identified from the audio frame;

[0248] Using the natural base as the base and the prediction score of each candidate molecule state as the index, the prediction index of each candidate molecule state is obtained.

[0249] The molecular prediction component of the audio frame is obtained by taking the logarithm of the sum of the prediction indices of at least one candidate molecular state.

[0250] In some embodiments, the second acquisition unit is used for:

[0251] From the target denominator lattice, query at least one candidate denominator state obtained from the identification of the audio frame;

[0252] Using the natural base as the base and the predicted score of each candidate denominator state as the exponent, obtain the predicted exponent for each candidate denominator state.

[0253] The denominator prediction component of the audio frame is obtained by taking the logarithm of the sum of the prediction exponents of at least one candidate denominator state.

[0254] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0255] It should be noted that the audio processing device provided in the above embodiments is only illustrated by the division of the above functional modules when recognizing the audio to be recognized. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the electronic device can be divided into different functional modules to complete all or part of the functions described above. In addition, the audio processing device and the audio processing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the audio processing method embodiments, which will not be repeated here.

[0256] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 10 As shown, the electronic device can be provided as terminal 1000. Optionally, the device type of terminal 1000 includes: smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 1000 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0257] Typically, terminal 1000 includes a processor 1001 and a memory 1002.

[0258] Optionally, the processor 1001 includes one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Optionally, the processor 1001 is implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). In some embodiments, the processor 1001 includes a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 1001 integrates a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 1001 also includes an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0259] In some embodiments, the memory 1002 includes one or more computer-readable storage media, optionally non-transitory. Optionally, the memory 1002 also includes high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 is used to store at least one program code, which is executed by the processor 1001 to implement the audio processing methods provided in the various embodiments of this application.

[0260] In some embodiments, the terminal 1000 may also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface 1003 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1003 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1008.

[0261] Peripheral device interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1001 and memory 1002. In some embodiments, processor 1001, memory 1002 and peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1001, memory 1002 and peripheral device interface 1003 are implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0262] The radio frequency (RF) circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. Optionally, the RF circuit 1004 communicates with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1004 also includes circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0263] Display screen 1005 is used to display a UI (User Interface). Optionally, the UI includes graphics, text, icons, videos, and any combination thereof. When display screen 1005 is a touch display screen, display screen 1005 also has the ability to collect touch signals on or above the surface of display screen 1005. The touch signals can be input to processor 1001 for processing as control signals. Optionally, display screen 1005 is also used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there is one display screen 1005, which is set on the front panel of terminal 1000; in other embodiments, there are at least two display screens 1005, which are respectively set on different surfaces of terminal 1000 or have a folded design; in some embodiments, display screen 1005 is a flexible display screen, which is set on the curved surface or folded surface of terminal 1000. Furthermore, optionally, display screen 1005 is set as a non-rectangular irregular shape, that is, an irregularly shaped screen. Optionally, the display screen 1005 is made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0264] The camera assembly 1006 is used to acquire images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1006 also includes a flash. Optionally, the flash is a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, used for light compensation at different color temperatures.

[0265] In some embodiments, the audio circuit 1007 includes a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 1001 for processing, or input to the radio frequency circuit 1004 to realize voice communication. For stereo acquisition or noise reduction purposes, multiple microphones are used, each located at a different part of the terminal 1000. Optionally, the microphone is an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert electrical signals from the processor 1001 or the radio frequency circuit 1004 into sound waves. Optionally, the speaker is a conventional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into sound waves that humans can hear, but also into sound waves that humans cannot hear for purposes such as distance measurement. In some embodiments, the audio circuit 1007 also includes a headphone jack.

[0266] The power supply 1008 is used to power the various components in the terminal 1000. Optionally, the power supply 1008 is AC power, DC power, a disposable battery, or a rechargeable battery. When the power supply 1008 includes a rechargeable battery, the rechargeable battery supports wired or wireless charging. The rechargeable battery also supports fast charging technology.

[0267] In some embodiments, the terminal 1000 further includes one or more sensors 1010. The one or more sensors 1010 include, but are not limited to: an acceleration sensor 1011, a gyroscope sensor 1012, a pressure sensor 1013, an optical sensor 1014, and a proximity sensor 1015.

[0268] In some embodiments, the accelerometer 1011 detects the magnitude of acceleration along the three coordinate axes of a coordinate system established with the terminal 1000. For example, the accelerometer 1011 is used to detect the components of gravitational acceleration along the three coordinate axes. Optionally, the processor 1001 controls the display screen 1005 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by the accelerometer 1011. The accelerometer 1011 is also used for acquiring motion data from games or the user.

[0269] In some embodiments, the gyroscope sensor 1012 detects the orientation and rotation angle of the terminal 1000. The gyroscope sensor 1012 and the accelerometer sensor 1011 work together to acquire the user's 3D movements on the terminal 1000. Based on the data acquired by the gyroscope sensor 1012, the processor 1001 implements the following functions: motion sensing (e.g., changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0270] Optionally, the pressure sensor 1013 is disposed on the side bezel of the terminal 1000 and / or on the lower layer of the display screen 1005. When the pressure sensor 1013 is disposed on the side bezel of the terminal 1000, it can detect the user's grip signal on the terminal 1000, and the processor 1001 performs left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1013. When the pressure sensor 1013 is disposed on the lower layer of the display screen 1005, the processor 1001 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1005. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0271] An optical sensor 1014 is used to collect ambient light intensity. In one embodiment, the processor 1001 controls the display brightness of the display screen 1005 based on the ambient light intensity collected by the optical sensor 1014. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is decreased. In another embodiment, the processor 1001 also dynamically adjusts the shooting parameters of the camera assembly 1006 based on the ambient light intensity collected by the optical sensor 1014.

[0272] The proximity sensor 1015, also known as a distance sensor, is typically installed on the front panel of the terminal 1000. The proximity sensor 1015 is used to detect the distance between the user and the front of the terminal 1000. In one embodiment, when the proximity sensor 1015 detects that the distance between the user and the front of the terminal 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from a screen-on state to a screen-off state; when the proximity sensor 1015 detects that the distance between the user and the front of the terminal 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from a screen-off state to a screen-on state.

[0273] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on terminal 1000, and it may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0274] Figure 11 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application, such as... Figure 11As shown, the electronic device 1100 can vary considerably due to differences in configuration or performance. The electronic device 1100 includes one or more Central Processing Units (CPUs) 1101 and one or more memories 1102. The memories 1102 store at least one computer program, which is loaded and executed by the one or more processors 1101 to implement the audio processing methods provided in the various embodiments described above. Optionally, the electronic device 1100 also includes wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The electronic device 1100 also includes other components for implementing device functions, which will not be elaborated upon here.

[0275] In exemplary embodiments, a computer-readable storage medium is also provided, such as a memory including at least one computer program, which can be executed by a processor in an electronic device to perform the audio processing methods in the various embodiments described above. For example, the computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0276] In an exemplary embodiment, a computer program product is also provided, including one or more computer programs stored in a computer-readable storage medium. One or more processors of an electronic device are capable of reading the one or more computer programs from the computer-readable storage medium, and the one or more processors execute the one or more computer programs, enabling the electronic device to perform the audio processing method described in the above embodiments.

[0277] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. Optionally, the program is stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0278] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An audio processing method, characterized in that, The method includes: Decode multiple audio frames in the audio to be identified to obtain a predicted word grid for the multiple audio frames. The predicted word grid provides a character sequence of multiple candidate texts identified from the multiple audio frames. Based on any candidate text from the plurality of audio frames, a word grid to be operated on is obtained, wherein the word grid to be operated on is used to provide the character sequence of the candidate text or the global character sequence of different candidate texts; An intersection operation is performed on the predicted word grid and the word grid to be operated on to obtain the target word grid and path relationship information. The path relationship information indicates the association relationship between each arc in the target word grid and the same arc in the word grid to be operated on. Each arc in the target word grid indicates a character. Based on the target word grid and the path relationship information, the prediction scores of multiple states in the target word grid are determined, and the prediction scores indicate the degree of semantic matching between each state and the preceding audio frame in the audio to be identified; Based on the prediction scores of multiple states in the target word lattice, the prediction probability of the candidate text is determined, and the prediction probability characterizes the degree of semantic matching between the candidate text and the audio to be identified.

2. The method according to claim 1, characterized in that, The step of determining the prediction scores of multiple states in the target word grid based on the target word grid and the path relationship information includes: Each arc in the target word grid is filtered to obtain multiple target arcs, so that each state in the target word grid has one and only one target arc that reaches that state; Based on the path relationship information, the associated arc of each target arc is determined from the word grid to be operated on, wherein the target arc and the associated arc indicate the same character in the word grid; Based on the forward score of the arrival state connected by each associated arc in the word lattice to be operated on, a predicted score of the arrival state in the target word lattice is assigned.

3. The method according to claim 2, characterized in that, The method further includes: Determine the arrival status of each associated arc connected in the word lattice to be operated on; Based on each arc in the word lattice that reaches the arrival state, obtain the forward score of the arrival state.

4. The method according to claim 3, characterized in that, The step of obtaining the forward score of the arrival state based on each arc in the word lattice that reaches the arrival state includes: For each arc in the word lattice that reaches the arrival state, the forward fraction of the starting state connected to each arc and the weight on each arc are added together to obtain the forward weight parameter of each arc. Using the natural base as the base and the forward weight parameter of each arc as the exponent, obtain the forward weight index of each arc; The forward score of the reached state is obtained by taking the logarithm of the sum of the forward weight indices of each arc.

5. The method according to claim 2, characterized in that, Assigning a predicted score to the target word lattice based on the forward score of the arrival state connected by each associated arc in the word lattice to be operated on includes: The query is performed on each state of the word to be operated on to obtain multiple candidate states that meet the query conditions. The query conditions indicate that there is an arc that starts from the current state and reaches the final state. The arc that reaches the final state indicates the character identified in the last audio frame among the multiple audio frames. If the arrival state belongs to the candidate state, the weight on the arc from the arrival state to the terminal state is added to the forward score of the arrival state to obtain the predicted score of the arrival state. If the arrival state does not belong to the candidate state, a preset value is assigned to the predicted score of the arrival state.

6. The method according to claim 1, characterized in that, The word grid to be operated on includes a numerator word grid and a denominator word grid. The numerator word grid provides the character sequence of the candidate text, and the denominator word grid provides the global character sequence of different candidate texts. The step of performing an intersection operation on the predicted word grid and the word grid to be operated on to obtain the target word grid includes: Perform an intersection operation on the predicted word lattice and the molecular word lattice to obtain the target molecular word lattice; The target denominator lattice is obtained by performing an intersection operation on the predicted lattice and the denominator lattice.

7. The method according to claim 6, characterized in that, The determination of the prediction probability of the candidate text based on the prediction scores of multiple states in the target word lattice includes: For each of the plurality of audio frames, based on the target molecular word lattice, obtain the molecular prediction component of the audio frame; Based on the target denominator lattice, obtain the denominator prediction component of the audio frame; Based on the numerator and denominator prediction components of the audio frame, the single-frame prediction component of the audio frame is determined. The prediction probability of the candidate text is obtained based on the single-frame prediction components of each of the multiple audio frames.

8. The method according to claim 7, characterized in that, The step of obtaining the molecular prediction components of the audio frame based on the target molecular lattice includes: From the target molecular lattice, query at least one candidate molecular state identified from the audio frame; Using the natural base as the base and the prediction score of each candidate molecule state as the index, the prediction index of each candidate molecule state is obtained. The molecular prediction component of the audio frame is obtained by taking the logarithm of the sum of the prediction indices of the at least one candidate molecular state.

9. The method according to claim 7, characterized in that, The step of obtaining the denominator prediction component of the audio frame based on the target denominator lattice includes: From the target denominator lattice, query at least one candidate denominator state obtained from the audio frame; Using the natural base as the base and the predicted score of each candidate denominator state as the exponent, obtain the predicted exponent for each candidate denominator state. The denominator prediction component of the audio frame is obtained by taking the logarithm of the sum of the prediction exponents of the at least one candidate denominator state.

10. An audio processing apparatus, characterized in that, The device includes: The decoding module is used to decode multiple audio frames in the audio to be identified, and obtain the predicted word grids of the multiple audio frames. The predicted word grids provide character sequences of multiple candidate texts identified from the multiple audio frames. The acquisition module is used to acquire a word grid to be operated on based on any candidate text of the plurality of audio frames. The word grid to be operated on is used to provide the character sequence of the candidate text or the global character sequence of different candidate texts. The intersection operation module is used to perform an intersection operation on the predicted word grid and the word grid to be operated on to obtain the target word grid and path relationship information. The path relationship information indicates the association relationship between each arc in the target word grid and the same arc in the word grid to be operated on. Each arc in the target word grid indicates a character. The score determination module is used to determine the predicted scores of multiple states in the target word grid based on the target word grid and the path relationship information. The predicted scores indicate the degree of semantic matching between each state and the preceding audio frame in the audio to be identified. The probability determination module is used to determine the prediction probability of the candidate text based on the prediction scores of multiple states in the target word lattice. The prediction probability characterizes the degree of semantic matching between the candidate text and the audio to be identified.

11. The apparatus according to claim 10, characterized in that, The score determination module includes: The filtering unit is used to filter each arc in the target word grid to obtain multiple target arcs, so that each state in the target word grid has one and only one target arc that reaches the state; An arc determination unit is used to determine, based on the path relationship information, the associated arc of each target arc from the word grid to be operated on, wherein the target arc and the associated arc indicate the same character in the word grid; The assignment unit is used to assign a value to the predicted score of the arrival state in the target word lattice based on the forward score of the arrival state connected by each associated arc in the word lattice to be operated.

12. The apparatus according to claim 11, characterized in that, The score determination module also includes: A state determination unit is used to determine the arrival state connected to each associated arc in the word lattice to be operated on; The score acquisition unit is used to acquire the forward score of the arrival state based on each arc in the word lattice that reaches the arrival state.

13. The apparatus according to claim 12, characterized in that, The score acquisition unit is used for: For each arc in the word lattice that reaches the arrival state, the forward fraction of the starting state connected to each arc and the weight on each arc are added together to obtain the forward weight parameter of each arc. Using the natural base as the base and the forward weight parameter of each arc as the exponent, obtain the forward weight index of each arc; The forward score of the reached state is obtained by taking the logarithm of the sum of the forward weight indices of each arc.

14. The apparatus according to claim 11, characterized in that, The assignment unit is used for: The query is performed on each state of the word to be operated on to obtain multiple candidate states that meet the query conditions. The query conditions indicate that there is an arc that starts from the current state and reaches the final state. The arc that reaches the final state indicates the character identified in the last audio frame among the multiple audio frames. If the arrival state belongs to the candidate state, the weight on the arc from the arrival state to the terminal state is added to the forward score of the arrival state to obtain the predicted score of the arrival state. If the arrival state does not belong to the candidate state, a preset value is assigned to the predicted score of the arrival state.

15. The apparatus according to claim 10, characterized in that, The word grid to be operated on includes a numerator word grid and a denominator word grid. The numerator word grid provides the character sequence of the candidate text, and the denominator word grid provides the global character sequence of different candidate texts. The intersection operation module is used for: Perform an intersection operation on the predicted word lattice and the molecular word lattice to obtain the target molecular word lattice; The target denominator lattice is obtained by performing an intersection operation on the predicted lattice and the denominator lattice.

16. The apparatus according to claim 15, characterized in that, The probability determination module includes: The first acquisition unit is used to acquire the molecular prediction component of each audio frame in the plurality of audio frames based on the target molecular word lattice. The second acquisition unit is used to acquire the denominator prediction component of the audio frame based on the target denominator word lattice; The prediction component determination unit is used to determine the single-frame prediction component of the audio frame based on the numerator prediction component and the denominator prediction component of the audio frame. The probability acquisition unit is used to acquire the prediction probability of the candidate text based on the single-frame prediction components of each of the multiple audio frames.

17. The apparatus according to claim 16, characterized in that, The first acquisition unit is used for: From the target molecular lattice, query at least one candidate molecular state identified from the audio frame; Using the natural base as the base and the prediction score of each candidate molecule state as the index, the prediction index of each candidate molecule state is obtained. The molecular prediction component of the audio frame is obtained by taking the logarithm of the sum of the prediction indices of the at least one candidate molecular state.

18. The apparatus according to claim 16, characterized in that, The second acquisition unit is used for: From the target denominator lattice, query at least one candidate denominator state obtained from the audio frame; Using the natural base as the base and the predicted score of each candidate denominator state as the exponent, obtain the predicted exponent for each candidate denominator state. The denominator prediction component of the audio frame is obtained by taking the logarithm of the sum of the prediction exponents of the at least one candidate denominator state.

19. An electronic device, characterized in that, The electronic device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the audio processing method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the audio processing method as described in any one of claims 1 to 9.

21. A computer program product, characterized in that, The computer program product includes at least one computer program, which is loaded and executed by a processor to implement the audio processing method as described in any one of claims 1 to 9.

Images