Speech coding method, speech coding apparatus, terminal device, and storage medium

Abstract

The application is suitable for the field of artificial intelligence technology, and provides a speech coding method, a speech coding device, a terminal device and a storage medium. The speech coding method comprises: coding a plurality of sorted speech data frames to obtain a feature frame sequence, wherein the plurality of speech data frames are obtained by performing a framing operation on to-be-identified speech data; calculating a blank probability of each feature frame in the feature frame sequence; determining an effective frame from the feature frame sequence based on the blank probability; determining a supplementary frame based on the position of the effective frame in the feature frame sequence; and coding the effective frame and the supplementary frame again to obtain coding data. The method can determine the effective frame and the supplementary frame as input data in the subsequent stage through a screening mechanism in the coding stage of speech recognition, thereby shortening the length of the feature frame sequence in the subsequent calculation process of speech recognition, reducing the calculation amount in the speech recognition process, and improving the calculation efficiency of speech recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech coding method, speech coding device, terminal equipment, and computer-readable storage medium. Background Technology

[0002] With the development of speech technology, speech coding technology has been widely applied in various aspects of life. For longer speech sequences, end-to-end speech coding methods have achieved results surpassing traditional methods. Although this speech coding method has significantly improved recognition accuracy, it still suffers from low computational efficiency. Summary of the Invention

[0003] In view of this, embodiments of this application provide a speech coding method, a speech coding device, a terminal device, and a computer-readable storage medium, which can shorten the length of the feature frame sequence, thereby reducing the amount of computation in the speech recognition process and improving the computational efficiency of speech recognition.

[0004] The first aspect of this application provides a speech coding method, including:

[0005] Encode multiple sorted speech data frames to obtain a feature frame sequence. The multiple speech data frames are obtained by performing a frame segmentation operation on the speech data to be recognized.

[0006] Calculate the blank probability of each feature frame in the above feature frame sequence;

[0007] Based on the aforementioned blank probability, valid frames are determined from the aforementioned feature frame sequence;

[0008] Supplementary frames are determined based on the positions of the aforementioned valid frames in the aforementioned feature frame sequence;

[0009] Based on the above valid frames and the above supplementary frames, the data is encoded again to obtain encoded data.

[0010] A second aspect of this application provides a speech coding apparatus, comprising:

[0011] The first encoding module is used to encode multiple sorted speech data frames to obtain a feature frame sequence. The multiple speech data frames are obtained by performing a frame segmentation operation on the speech data to be recognized.

[0012] The calculation module is used to calculate the blank probability of each feature frame in the above feature frame sequence;

[0013] The first determining module is used to determine the valid frame from the above feature frame sequence based on the above blank probability;

[0014] The second determining module is used to determine the supplementary frame based on the position of the above-mentioned valid frame in the above-mentioned feature frame sequence;

[0015] The second encoding module is used to encode the above-mentioned valid frames and supplementary frames again to obtain encoded data.

[0016] A third aspect of this application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the terminal device. When the processor executes the computer program, it implements the steps of the speech coding method provided in the first aspect.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the speech coding method provided in the first aspect.

[0018] The speech coding method, speech coding apparatus, terminal device, and computer-readable storage medium provided in this application have the following beneficial effects:

[0019] By encoding multiple sorted speech data frames, each speech data frame yields a feature frame after encoding. Multiple speech data frames correspond to multiple feature frames, which, when arranged in temporal order, form a feature frame sequence. For each feature frame, a blank probability is calculated, and its validity is determined based on this probability. After identifying all valid frames in the feature frame sequence, supplementary frames are further determined based on their positions within the sequence. These supplementary frames are considered to potentially contain relevant information related to the valid frames. Finally, the valid and supplementary frames are re-encoded to obtain the encoded data. This method shortens the feature frame sequence length, reduces computational load in subsequent speech recognition processes, and thus improves the computational efficiency of speech recognition. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the implementation of a speech coding method provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the structure of an existing coding model provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of an improved coding model structure provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of an FFM network provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of an extractor provided in an embodiment of this application;

[0026] Figure 6 This is a structural block diagram of a speech coding device provided in an embodiment of this application;

[0027] Figure 7 This is a structural block diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] The voice encoding method involved in the embodiments of this application can be executed by a terminal device, such as a mobile phone, laptop computer, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA).

[0030] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0031] Foundational artificial intelligence technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning. BenShen's speech processing technology filters and selects from the encoded feature frame sequence, retaining valid and supplementary frames, thereby shortening the length of the feature frame sequence. This helps reduce the computational load in speech recognition and improves its efficiency.

[0032] Please see Figure 1 , Figure 1 A flowchart illustrating the implementation of a speech coding method provided in an embodiment of this application is shown. The speech coding method includes:

[0033] Step 110: Encode the sorted multiple speech data frames to obtain a feature frame sequence.

[0034] Because speech signals have short-term stationarity, they can be segmented for feature extraction to facilitate subsequent processing. Each segment is called a frame, also known as frame segmentation. After obtaining the speech data to be recognized, this data can be segmented into frames to obtain multiple speech data frames. Since speech signals have temporal characteristics, these frames can be sorted according to their time sequence to obtain sorted speech data frames. Each speech data frame is then encoded to obtain a corresponding feature frame. Finally, by arranging the feature frames corresponding to each speech data frame in temporal order (i.e., according to the order of the corresponding speech data frames), a feature frame sequence is obtained.

[0035] Step 120: Calculate the blank probability of each feature frame in the feature frame sequence.

[0036] Connectionist Temporal Classification (CTC) is a classification method that avoids manual alignment of input and output. Considering the characteristics of the sequences obtained based on this method and the output independence assumption of CTC, this application proposes two assumptions: First, for feature frames corresponding to non-blank labels in the output, if they contain enough information to distinguish the label corresponding to that frame, such feature frames are recorded as valid frames; second, for feature frames corresponding to blank labels in the output, if they contain less information to distinguish the label corresponding to that frame, such feature frames are recorded as blank frames. To distinguish which feature frames are valid frames and which are blank frames, the blank probability of each feature frame can be calculated, thereby distinguishing between these two types of feature frames in the feature frame sequence.

[0037] Step 130: Determine the valid frames from the feature frame sequence based on the blank probability.

[0038] To reduce the computational load of speech recognition, blank frames can be removed from the feature frame sequence, retaining only valid frames. This shortens the feature frame sequence used for subsequent calculations, thereby improving the computational efficiency of speech recognition. Specifically, after calculating the blank probability of each feature frame, it can be determined whether the feature frame is a valid frame based on this blank probability.

[0039] Step 140: Determine the supplementary frame based on the position of the valid frame in the feature frame sequence.

[0040] To avoid losing relevant information during the determination of valid frames, supplementary frames can be determined based on their position in the feature frame sequence. For example, valid frames can be categorized according to their blankness probability. A higher blankness probability corresponds to a lower level; higher-level blank frames have more supplementary frames to choose from, and vice versa. It's understood that supplementary frames are selected from blank frames.

[0041] Step 150: Encode again based on the valid frame and the supplementary frame to obtain the encoded data.

[0042] After filtering the feature frame sequence, the valid frames and supplementary frames in the feature frame sequence are obtained. The order of the valid frames and supplementary frames is the same as their order in the feature frame sequence. After re-encoding the valid frames and supplementary frames, the encoded data is obtained. This encoded data is the data obtained after filtering out blank frames containing very little effective information. Compared with the feature frame sequence obtained in step 110, the sequence length of this encoded data is effectively reduced. Performing subsequent speech recognition operations based on this encoded data can reduce the computational load of speech recognition and improve the efficiency of speech recognition.

[0043] By encoding multiple sorted speech data frames, each speech data frame yields a feature frame, and multiple speech data frames result in multiple feature frames. These feature frames, arranged in temporal order, form a feature frame sequence. For each feature frame, a blank probability is calculated, and based on this probability, it is determined whether the feature frame is a valid frame. After identifying all valid frames in the feature frame sequence, supplementary frames can be determined based on their positions within the sequence. These supplementary frames are considered to potentially contain relevant information related to the valid frames. Finally, the valid frames and supplementary frames are re-encoded to obtain the encoded data. This method shortens the feature frame sequence length, reduces the computational load in speech recognition, and improves the computational efficiency of speech recognition.

[0044] In some embodiments, in order to accurately determine whether each feature frame is a valid frame, step 120 specifically includes:

[0045] For each feature frame:

[0046] Step 121: Determine whether the blank probability of the feature frame is less than the preset first blank probability threshold.

[0047] Step 122: If the blank probability of the feature frame is less than the preset first blank probability threshold, then the feature frame is determined as a valid frame.

[0048] A blank probability threshold can be predetermined based on empirical values, and whether the blank probability is less than the blank probability threshold is used as the criterion for determining whether each feature frame is a valid frame. For each feature frame, the lower the blank probability, the more valid information it contains. If the blank probability of a feature frame is less than the blank probability threshold, the feature frame is determined to be a valid frame. By judging each feature frame in the feature sequence using this method, all valid frames in the sequence can be obtained. The blank probability threshold in this embodiment is used to filter valid frames. To distinguish it from another blank probability threshold mentioned later, this blank probability threshold is referred to as the first blank probability threshold, and the blank probability threshold mentioned later is referred to as the second blank probability threshold.

[0049] In some embodiments, to reduce the risk of losing valid information during the determination of valid frames, step 140 specifically includes:

[0050] If a target feature frame exists to the left and / or right of a valid frame in the feature frame sequence, then the target feature frame is determined as a supplementary frame.

[0051] To prevent some feature frames containing valid information from being filtered out, the blank probability threshold can be increased. A second blank probability threshold can be set to filter blank frames, and blank frames with a blank probability less than the second blank probability threshold are identified as supplementary frames to improve the integrity of valid information. The concepts of "left" and "right" of a valid frame are based on the position of the valid frame in the feature frame sequence. Assuming a segment of a feature frame sequence is "i, i+1, i+2", where i+1 is a valid frame, then the blank frame to the left of the valid frame is i, and the blank frame to the right of the valid frame is i+2.

[0052] In some embodiments, in addition to the method of secondary filtering of blank frames using a second blank probability threshold described above, step 140 can also be implemented through the following steps:

[0053] Step 141: Obtain each label of each valid frame and the corresponding output probability of each label;

[0054] Step 142: If there are two adjacent valid frames with the same target label in the feature frame sequence, then the feature frame between the two adjacent valid frames is determined as a supplementary frame.

[0055] For each feature frame, a labeling operation can be performed using an indicator, assigning different labels to the feature frame. Each label corresponds to a probability of outputting that label, i.e., an output probability. That is, if a feature frame corresponds to multiple output labels, each label has an output probability, and the label with the highest output probability is the target label for that feature frame. For example, if feature frame 'a' has 3 labels, then there are 3 output probabilities corresponding to these 3 labels. Assuming the output probabilities corresponding to the 3 labels are 65%, 55%, and 80%, then the label corresponding to the 80% output probability is the target label. In the feature frame sequence, if two adjacent valid frames have the same label with the highest output probability, that is, the target labels of these two valid frames are the same, then these two valid frames contain the same valid information. Therefore, the blank frames between the two valid frames also contain the same valid information and can be identified as supplementary frames to improve the completeness of the valid information. The term "adjacent valid frames" here refers to the feature frame sequence itself. After all valid frames in the feature frame sequence have been determined, the position of each valid frame within the feature frame sequence can be identified. In this feature sequence, if there is only a blank frame between two valid frames, then the positional relationship between these two valid frames is adjacent.

[0056] In some embodiments, in addition to the two methods for determining supplementary frames described above, step 140 can also be implemented through the following steps:

[0057] (1) Randomly select m frames (m<=n) from the n frames to the left of the valid frame. The closer the frame is to the valid frame, the greater the probability that it will be identified as a supplementary frame.

[0058] (2) Randomly select m frames (m<=n) from the n frames to the right of the valid frame. The closer the frame is to the valid frame, the greater the probability that it will be identified as a supplementary frame.

[0059] (3) Randomly select m frames (m<=n) from the n frames on both sides of the valid frame. The closer the frame is to the valid frame, the greater the probability that it will be identified as a supplementary frame.

[0060] After the valid frames in the feature frame sequence are determined, their positions within the sequence can be identified. Then, a specified number of blank frames are selected from the left and / or right sides of the valid frames according to a certain probability distribution. These selected blank frames are denoted as candidate frames; the closer they are to valid frames, the greater their probability of being identified as supplementary frames. Specifically, a distance threshold can be set. The distance between each candidate frame and the valid frame is first calculated. Then, each distance (the distance between a candidate frame and the nearest valid frame) is compared with the distance threshold. Candidate frames with distances less than the threshold are identified as supplementary frames.

[0061] In some embodiments, to further reduce the risk of losing valid information, after step 140, the method further includes:

[0062] A. Based on a preset extractor, feature information is extracted from the valid frame, supplementary frame, and feature frame sequence.

[0063] Even in blank frames, some valid information may still remain. To reduce the risk of losing this valid information, feature extraction can be performed based on valid frames, supplementary frames, and blank frames. Since extracting feature information from blank frames is quite difficult in practice, the possibility of extracting information from blank frames can be enhanced by expanding the extraction scope to the entire feature frame sequence. Specifically, feature frame sequence extraction (equivalent to extracting feature information from blank frames) can be performed simultaneously with valid frames and supplementary frames, thereby further reducing the risk of losing valid information and improving the completeness of feature information.

[0064] Accordingly, step 150 specifically includes: re-encoding based on feature information to obtain encoded data.

[0065] Accordingly, during the re-encoding process, the feature information obtained by the extractor can be directly encoded.

[0066] To further understand the speech coding method of this application, the model implementing this application is introduced. (See also...) Figure 2 , Figure 2 The diagram illustrates a classic speech coding model in the prior art. Clearly, this model lacks the filtering and extraction mechanisms described in this application. To shorten the encoded feature frame sequence, this application introduces a filtering and extraction mechanism, implemented using an FFM network. The FFM network can be embedded within a classic speech coding model; its specific structure is described in [reference needed]. Figure 3 , Figure 3 The improved encoding model obtained in this application is shown. This model can introduce filtering and extraction mechanisms during the encoding process to reduce the length of the feature frame sequence obtained after encoding, thereby reducing the computational load in subsequent post-processing of speech recognition and improving the computational efficiency of speech recognition.

[0067] Specifically, such as Figure 4 As shown, the FMM network can be embedded into the Conformer block. This is only an example and not a limitation. Assuming that the Conformer block has 15 layers, the FMM network can be embedded after the 7th layer. That is, the first 7 layers perform the first encoding, and the last 8 layers perform the second encoding on the feature information filtered and extracted by the FMM network to obtain the encoded data.

[0068] Specifically, see Figure 4 , Figure 4The diagram shows the structure of an FFM network, which includes an indicator and an extractor. The indicator is used to filter and determine the valid frames and supplementary frames in the feature frame sequence, and the extractor is used to extract the feature information from the valid frames, supplementary frames, and feature frame sequence (actually extracting the feature information from the blank frames).

[0069] In some embodiments, the extractor includes: a first feed forward net (FFN) module, a multi-head encoder-decoder attention (MHEDA) module, a convolution (CONV) module, a second feed forward net module, and a layer normalization (LN) module.

[0070] The first feedforward neural network module is used to perform a fully connected operation on the effective frame, the supplementary frame, and the feature frame sequence to obtain the first feature frame sequence.

[0071] The multi-head encoder-decoder attention module is used to perform self-attention encoder-decoder operations on the first feature frame sequence to obtain the second feature frame sequence.

[0072] The convolution module is used to perform convolution operations on the second feature frame sequence to obtain the third feature frame sequence.

[0073] The second feedforward neural network module is used to perform a fully connected operation on the third feature frame sequence to obtain the fourth feature frame sequence.

[0074] The normalization module is used to perform normalization operations on the fourth feature frame sequence to obtain feature information.

[0075] The mathematical expression for the extractor is as follows:

[0076] (1)

[0077] (2)X eda =X ffn +MHEDA(X ffn X e X e )

[0078] (3)X conv =X eda +CONV(X eda )

[0079] (4)

[0080] Among them, X fThese are the valid frames and supplementary frames sorted according to the feature frame sequence; X ffn The first feature frame sequence; X eda The second feature frame sequence; X conv The third feature frame sequence; Y e This refers to feature information.

[0081] See Figure 5 The above expression will be explained with reference to the structural diagram of the extractor:

[0082] Assume the feature frame sequence is X e X e After filtering, we can obtain the valid frames and supplementary frames X, sorted according to the feature frame sequence. f X e With X f In the input extractor, feature information is obtained through the following steps:

[0083] Step 1: X f After inputting the first feedforward neural network FNN, the output sequence FNN_1(X) can be obtained. f ), will FFN_1(X f Multiply by 0.5 to get the residual sum X f By summing the results, we obtain the first feature frame sequence X. ffn .

[0084] Step 2: Place X ffn The input sequence Xe is fed into the MHEDA module to calculate the attention distribution, resulting in the output sequence MHEDA(Xe). ffn X e X e ), MHEDA(X ffn X e X e ) will be used as the residual and X ffn Sum the results to obtain the second feature frame sequence X. eda .

[0085] Step 3: Place X eda The input is fed into the convolution module to obtain Conv(X) eda ), will Conv(X eda ) as residuals and X eda Sum, and the third feature frame sequence X is obtained. conv .

[0086] Step 4: Place X conv As input, it enters the second feedforward neural network module, resulting in FFN_2(X). conv ), FFN_2(X conv ) as residuals and X convAfter summation, the feature information Y is obtained by normalization layer regularization. e .

[0087] The specific expression for the MHEDA model is as follows:

[0088] (5)

[0089] (6)MHEDA(Q,K,V)=Concat(head1,...,head n W o

[0090] (7)

[0091] Where Q is the query, representing the sequence of feature frames to be queried; K is the key, representing the keyword; V is the value, representing the value sequence; and head... i This represents the i-th head.

[0092] In some embodiments, the above speech coding method further includes:

[0093] Deploy the feature frame sequence, valid frames, supplementary frames, multiple voice data frames and / or encoded data into the blockchain.

[0094] To ensure data security and fairness and transparency for users, feature frame sequences, valid frames, supplementary frames, multiple voice data frames, and / or encoded data can be deployed to the blockchain for evidence storage. Users can then download the feature frame sequences, valid frames, supplementary frames, multiple voice data frames, and / or encoded data from the blockchain using their respective devices to verify whether the data has been tampered with. The blockchain referred to in this embodiment is a novel application model employing computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0095] In addition, this application also provides a speech coding device.

[0096] Please see Figure 6 , Figure 6 This is a structural block diagram of a speech coding device provided in an embodiment of this application. In this embodiment, the terminal device includes units used for performing... Figure 1 The steps in the corresponding embodiments. Please refer to the details. Figure 1 as well as Figure 1The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 6 The speech coding device 60 includes:

[0097] The first encoding module 61 is used to encode multiple sorted speech data frames to obtain a feature frame sequence. The multiple speech data frames are obtained by performing a frame division operation on the speech data to be recognized.

[0098] The calculation module 62 is used to calculate the blank probability of each feature frame in the feature frame sequence;

[0099] The first determining module 63 is used to determine the valid frame from the feature frame sequence based on the blank probability;

[0100] The second determining module 64 is used to determine the supplementary frame based on the position of the valid frame in the feature frame sequence;

[0101] The second encoding module 65 is used to encode the valid frame and the supplementary frame again to obtain encoded data.

[0102] As an embodiment of this application, the first determining module 63 described above may include:

[0103] For each feature frame:

[0104] The judgment unit is used to determine whether the blank probability of the feature frame is less than a preset first blank probability threshold.

[0105] The first determining unit is used to determine the feature frame as a valid frame if the blank probability of the feature frame is less than the first blank probability threshold.

[0106] As an embodiment of this application, the second determining module 64 described above may include:

[0107] The second determining unit is used to determine the target feature frame as a supplementary frame if there is a target feature frame to the left and / or right of the effective frame in the feature frame sequence. The target feature frame is a feature frame in the feature frame sequence whose blank probability is less than a preset second blank probability threshold.

[0108] As an embodiment of this application, the second determining module 64 described above may include:

[0109] The acquisition unit is used to acquire each label of each valid frame and the corresponding output probability of each label;

[0110] The third determining unit is used to determine the feature frame between two adjacent valid frames as a supplementary frame if, in the feature frame sequence, there are two adjacent valid frames with the same target label, and the target label is the label with the highest output probability among all the labels of the valid frames.

[0111] As one embodiment of this application, the above-mentioned speech encoding device 60 may include:

[0112] The extraction module is used to extract feature information from the valid frame, the supplementary frame and the feature frame sequence based on a preset extractor after determining the supplementary frame based on the position of the valid frame in the feature frame sequence.

[0113] Accordingly, the second encoding module is specifically used to re-encode based on feature information to obtain encoded data.

[0114] As one embodiment of this application, the above-mentioned speech coding device 60 further includes:

[0115] The deployment module is used to deploy feature frame sequences, valid frames, supplementary frames, multiple voice data frames, and / or encoded data to the blockchain.

[0116] Figure 7 This is a structural block diagram of a terminal device provided in another embodiment of this application. For example... Figure 7 As shown, the terminal device 70 of this embodiment includes: a processor 71, a memory 72, and a computer program 73 stored in the memory 72 and executable on the processor 71, such as a program for a speech coding method. When the processor 71 executes the computer program 73, it implements the steps of each embodiment of the various speech coding methods described above, for example... Figure 1 As shown in 110 to 150. Alternatively, the processor 71 executes the computer program 73 to implement the above. Figure 6 The functions of each module in the corresponding embodiments, for example, Figure 6 For details on the functions of modules 61 to 65 shown, please refer to [link / reference needed]. Figure 6 The relevant descriptions in the corresponding embodiments are not repeated here.

[0117] For example, the computer program 73 described above can be divided into one or more units, which are stored in the memory 72 and executed by the processor 71 to complete this application. The one or more units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 73 in the terminal 70. For example, the computer program 73 can be divided into a first encoding module, a calculation module, a first determining module, a second determining module, and a second encoding module, with the specific functions of each module as described above.

[0118] The aforementioned terminal device may include, but is not limited to, a processor 71 and a memory 72. Those skilled in the art will understand that... Figure 7This is merely an example of terminal device 70 and does not constitute a limitation on terminal device 70. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device described above may also include input / output devices, network access devices, buses, etc.

[0119] The processor 71 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0120] The aforementioned memory 72 can be an internal storage unit of the terminal device 70, such as a hard disk or RAM of the terminal device 70. The aforementioned memory 72 can also be an external storage device of the terminal device 70, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 70. Furthermore, the aforementioned memory 72 can include both internal and external storage units of the terminal device 70. The aforementioned memory 72 is used to store the aforementioned computer program and other programs and data required by the terminal device. The aforementioned memory 72 can also be used to temporarily store data that has been output or will be output.

[0121] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A speech coding method, characterized in that, The speech coding method includes: Encode multiple sorted speech data frames to obtain a feature frame sequence, wherein the multiple speech data frames are obtained by performing a frame segmentation operation on the speech data to be identified; The blank probability of each feature frame in the feature frame sequence is calculated based on the CTC framework; Valid frames are determined from the feature frame sequence based on the blank probability; Supplementary frames are determined from the invalid frames based on the position of the valid frames in the feature frame sequence; Encode the valid frame and the supplementary frame again to obtain encoded data; After determining the supplementary frame based on the position of the effective frame in the feature frame sequence, the speech coding method further includes: Feature information is extracted from the valid frame, the supplementary frame, and the feature frame sequence based on a preset extractor; Accordingly, the step of re-encoding based on the valid frame and the supplementary frame to obtain encoded data includes: The encoded data is obtained by re-encoding based on the aforementioned feature information; The extractor includes: a first feedforward neural network module, a multi-head encoder-decoder attention module, a convolution module, a second feedforward neural network module, and a normalization module; The first feedforward neural network module is used to perform a fully connected operation on the effective frame, the supplementary frame, and the feature frame sequence to obtain the first feature frame sequence; The multi-head encoder-decoder attention module is used to perform self-attention encoder-decoder operations on the first feature frame sequence to obtain the second feature frame sequence; The convolution module is used to perform a convolution operation on the second feature frame sequence to obtain the third feature frame sequence; The second feedforward neural network module is used to perform a fully connected operation on the third feature frame sequence to obtain the fourth feature frame sequence. The normalization module is used to perform a normalization operation on the fourth feature frame sequence to obtain the feature information.

2. The speech coding method according to claim 1, characterized in that, The step of determining valid frames from the feature frame sequence based on the blank probability includes: For each feature frame: Determine whether the blank probability of the feature frame is less than a preset first blank probability threshold; If the blank probability of the feature frame is less than the first blank probability threshold, then the feature frame is determined to be a valid frame.

3. The speech coding method according to claim 1, characterized in that, Determining the supplementary frame based on the position of the valid frame in the feature frame sequence includes: If a target feature frame exists to the left and / or right of the valid frame in the feature frame sequence, then the target feature frame is determined as the supplementary frame. The target feature frame is a feature frame in the feature frame sequence whose blank probability is less than a preset second blank probability threshold.

4. The speech coding method according to claim 1, characterized in that, Determining the supplementary frame based on the position of the valid frame in the feature frame sequence includes: Obtain each label of each valid frame and the output probability corresponding to each label; If, in the feature frame sequence, there are two adjacent valid frames with the same target label, and the target label is the label with the highest output probability among all the labels of the valid frames, then the feature frame between the two adjacent valid frames is determined as a supplementary frame.

5. The speech coding method according to claim 1, characterized in that, After encoding the data again based on the valid frame and the supplementary frame to obtain encoded data, the speech encoding method further includes: The feature frame sequence, the valid frame, the supplementary frame, the plurality of the voice data frames and / or the encoded data are deployed to the blockchain.

6. A voice encoding device, characterized in that, The speech encoding device includes: The first encoding module is used to encode multiple sorted speech data frames to obtain a feature frame sequence, wherein the multiple speech data frames are obtained by performing a frame segmentation operation on the speech data to be recognized. The calculation module is used to calculate the blank probability of each feature frame in the feature frame sequence based on the CTC framework; The first determining module is used to determine valid frames from the feature frame sequence based on the blank probability; The second determining module is used to determine a supplementary frame from the invalid frames based on the position of the valid frame in the feature frame sequence; The second encoding module is used to encode the valid frame and the supplementary frame again to obtain encoded data; The speech encoding device further includes: The extraction module is used to extract feature information from the effective frame, the supplementary frame, and the feature frame sequence based on a preset extractor; Accordingly, the second encoding module is specifically used to re-encode the feature information to obtain encoded data; The extractor includes: a first feedforward neural network module, a multi-head encoder-decoder attention module, a convolution module, a second feedforward neural network module, and a normalization module; The first feedforward neural network module is used to perform a fully connected operation on the effective frame, the supplementary frame, and the feature frame sequence to obtain the first feature frame sequence; The multi-head encoder-decoder attention module is used to perform self-attention encoder-decoder operations on the first feature frame sequence to obtain the second feature frame sequence; The convolution module is used to perform a convolution operation on the second feature frame sequence to obtain the third feature frame sequence; The second feedforward neural network module is used to perform a fully connected operation on the third feature frame sequence to obtain the fourth feature frame sequence. The normalization module is used to perform a normalization operation on the fourth feature frame sequence to obtain the feature information.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images