Voice processing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a voice processing method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining an input voice feature sequence, and dividing the voice feature sequence to obtain a plurality of adjacent feature blocks; performing context modeling on each feature block to extract context information in each feature block; establishing a connection relationship between adjacent feature blocks based on a cross-block attention mechanism; encoding the context information of each feature block, and grouping the encoded context information based on a preset clustering model to generate a discretized voice result; and generating an output voice based on a preset large voice model, the discretized voice result and a connection relationship between adjacent feature blocks. The technical problem that dynamic real-time information cannot be adapted in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and particularly to a voice processing method, device, computer device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, spoken dialogue systems have been widely used in fields such as intelligent customer service, voice assistants, and smart homes.

[0003] In the field of medical and health, a spoken dialogue system can be deployed in an intelligent consultation robot. Patients input their conditions by voice, and the intelligent consultation robot recognizes the voice input by the patients, and then answers the patients' questions, which can replace some of the work of doctors, relieve the pressure on doctors, and also provide convenience for users.

[0004] In the field of fintech business, a spoken dialogue system can be deployed in an intelligent customer service. The intelligent customer service can solve the problems of patients according to the voice input by the patients, and while facilitating users, reduce the work pressure of human customer service.

[0005] However, the existing end-to-end spoken dialogue systems have obvious defects and deficiencies in the following aspects;

[0006] 1. Limited dialogue understanding and generation capabilities. Existing systems usually adopt a cascaded mode of ASR, NLP, and TTS. There is information fragmentation between these modules, resulting in insufficient coherence in dialogue understanding and generation. In addition, when dealing with complex dialogue scenarios, existing systems often have difficulty accurately understanding the user's intention. Especially in multi-turn conversations, the relevance of the dialogue context is weak, and logical breaks are likely to occur.

[0007] 2. Limited understanding ability for long voice segments. The current voice generation model is limited by the short sequence modeling ability and cannot effectively process long voice segments. The traditional attention mechanism needs to perform global calculations on the entire sequence, resulting in a quadratic growth in the amount of calculation and making it difficult to process long context information. This limits its application in actual scenarios. For example, scenarios such as long conversations and long text-to-speech require the model to be able to process longer context information. Summary of the Invention

[0008] The purpose of the present invention is to overcome the above technical deficiencies, and provide a voice processing method, device, computer device and storage medium to solve the technical problem of limited dialogue understanding ability of the spoken dialogue system in the prior art.

[0009] To achieve the above technical purpose, the present invention adopts the following technical solutions:

[0010] In the first aspect, the present invention provides a voice processing method, including the following steps:

[0011] Obtain the input speech feature sequence, divide the speech feature sequence to obtain a plurality of adjacent feature blocks;

[0012] Perform context modeling on each of the feature blocks to extract context information within each feature block;

[0013] Based on the cross-block attention mechanism, establish connection relationships between adjacent feature blocks;

[0014] Encode the context information of each of the feature blocks, and group the encoded context information based on a preset clustering model to generate a discretized speech result;

[0015] Generate an output speech based on a preset large speech model, the discretized speech result, and the connection relationships between adjacent feature blocks.

[0016] In some embodiments, the obtaining the input speech feature sequence, dividing the speech feature sequence to obtain a plurality of adjacent feature blocks includes:

[0017] Obtain the input speech feature sequence, and parse out the time length corresponding to the speech feature sequence to obtain the time block corresponding to the speech feature sequence;

[0018] Based on the preset size of the time division block, divide the time block corresponding to the speech feature sequence;

[0019] Based on the divided time blocks, divide the speech feature sequence to obtain a plurality of adjacent feature blocks.

[0020] In some embodiments, the based on the preset size of the time division block, dividing the time block corresponding to the speech feature sequence includes:

[0021] Judge whether the size of the time block corresponding to the speech feature sequence exceeds a preset value;

[0022] When the size of the time block corresponding to the speech feature sequence does not exceed the preset value, divide the time block corresponding to the speech feature sequence based on the preset size of the time division block;

[0023] When the size of the time block corresponding to the speech feature sequence exceeds the preset value, after increasing the preset size of the time division block, divide the time block corresponding to the speech feature sequence based on the increased preset size of the time division block;

[0024] Perform boundary adjustment and feature smoothing processing on the divided time blocks in sequence to obtain an optimized feature block sequence.

[0025] In some embodiments, context modeling of each of the feature blocks to extract context information within each feature block includes:

[0026] Based on a preset self-attention mechanism, calculate the first attention weights between each element within the feature block and other elements;

[0027] Based on the calculated first attention weights, perform weighted summation on each element within the feature block to obtain the context information of each feature block;

[0028] Perform feature compression, feature enhancement, and feature normalization processing on the context information in sequence to generate optimized feature block context information.

[0029] In some embodiments, establishing connection relationships between adjacent feature blocks based on a cross-block attention mechanism includes:

[0030] Based on a preset cross-block attention mechanism, calculate the second attention weights between each feature block and other feature blocks;

[0031] Based on the calculated second attention weights, perform weighted summation on each of the feature blocks to obtain the connection relationships between adjacent feature blocks;

[0032] Perform feature enhancement, feature normalization, and feature smoothing processing on the connection relationships in sequence to enhance the stability and consistency of the connection relationships.

[0033] In some embodiments, encoding the context information of each of the feature blocks, and grouping the encoded context information based on a preset clustering model to generate a discretized speech result includes:

[0034] Adopt a preset encoding model to encode the context information of each of the feature blocks;

[0035] Group the encoded context information based on a preset clustering model to generate a discretized speech result;

[0036] Perform feature enhancement and feature normalization processing on the discretized speech result in sequence, and verify the speech result after normalization processing;

[0037] According to the verification result, perform optimization adjustment on the speech result after normalization to generate a discretized speech feature sequence.

[0038] In some embodiments, generating an output speech based on a preset large speech model, the discretized speech result, and the connection relationships between adjacent feature blocks includes:

[0039] Input the discretized speech result into a preset large language model to generate a speech representation;

[0040] Fuse the discretized speech result and the speech representation to generate a speech output result;

[0041] Generate an output speech based on the connection relationship between adjacent feature blocks and the speech output result.

[0042] In a second aspect, the present invention also provides a speech processing device, including:

[0043] A partitioning module, configured to obtain an input speech feature sequence, partition the speech feature sequence to obtain a plurality of adjacent feature blocks;

[0044] A context information extraction module, configured to perform context modeling on each of the feature blocks to extract context information within each feature block;

[0045] A connection relationship establishment module, configured to establish a connection relationship between adjacent feature blocks based on a cross-block attention mechanism;

[0046] An encoding module, configured to encode the context information of each of the feature blocks, group the encoded context information based on a preset clustering model to generate a discretized speech result;

[0047] An output module, configured to generate an output speech based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks.

[0048] In a third aspect, the present invention also provides a computer device, including a memory and a processor, where computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, the steps of the speech processing method described above are implemented.

[0049] In a fourth aspect, the present invention also provides a computer-readable storage medium, where computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the steps of the speech processing method described above are implemented.

[0050] Compared with the prior art, the voice processing method, device, computer device and storage medium provided by the present invention first obtain an input voice feature sequence, divide the voice feature sequence to obtain a plurality of adjacent feature blocks; then perform context modeling on each of the feature blocks to extract context information within each feature block; then based on a cross-block attention mechanism, establish a connection relationship between adjacent feature blocks; then encode the context information of each of the feature blocks, and group the encoded context information based on a preset clustering model to generate a discretized voice result; finally, based on a preset large voice model, the discretized voice result and the connection relationship between adjacent feature blocks, generate an output voice. By introducing a block attention mechanism and a more robust clustering algorithm, the present invention significantly improves the computational efficiency, robustness and expressiveness of the model. The block attention mechanism significantly reduces the computational complexity and improves the long context processing ability, and the more robust clustering algorithm enhances the accuracy and stability of voice discretization, improves the semantic retention effect, can better process multi-language voice data and complex noise scenarios, and improves the versatility and adaptability of the model. In addition, the present invention also provides a solid technical foundation for the popularization of voice synthesis technology in applications such as multi-languages, complex scenarios, and long voice inputs. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the solutions in the present invention, the following will briefly introduce the drawings required for the description of the embodiments of the present invention. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0052] Figure 1 is an exemplary system architecture diagram to which the present invention can be applied;

[0053] Figure 2 is a flowchart of an embodiment of the voice processing method according to the present invention;

[0054] Figure 3 is Figure 2 a flowchart of a specific embodiment of step S100 shown;

[0055] Figure 4 is Figure 2 a flowchart of a specific embodiment of step S200 shown;

[0056] Figure 5 is Figure 2 a flowchart of a specific embodiment of step S300 shown;

[0057] Figure 6 is Figure 2Flowchart of a specific embodiment of step S400 shown;

[0058] Figure 7 is Figure 2 Flowchart of a specific embodiment of step S500 shown;

[0059] Figure 8 Schematic structural diagram of an embodiment of a voice processing device according to the present invention;

[0060] Figure 9 Schematic structural diagram of an embodiment of a computer device according to the present invention;

[0061] Figure 10 Schematic structural diagram of another embodiment of a computer device according to the present invention. Detailed implementation manners

[0062] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] Referring to "embodiments" herein means that specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0064] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0065] The voice processing method based on artificial intelligence provided by the embodiments of the present invention can be applied in such as Figure 1In the application environment, the client communicates with the server through the network. The server can obtain the input speech feature sequence through the client, divide the speech feature sequence to obtain multiple adjacent feature blocks; perform context modeling on each feature block to extract the context information within each feature block; establish connection relationships between adjacent feature blocks based on the cross-block attention mechanism; encode the context information of each feature block, and group the encoded context information based on a preset clustering model to generate a discretized speech result; generate an output speech based on a preset large speech model, the discretized speech result, and the connection relationships between adjacent feature blocks. By introducing the block attention mechanism and a more robust clustering algorithm, the present invention significantly improves the computational efficiency, robustness, and expressiveness of the model. The block attention mechanism significantly reduces the computational complexity and improves the long-context processing ability. The more robust clustering algorithm enhances the accuracy and stability of speech discretization, improves the semantic retention effect, can better process multi-lingual speech data and complex noise scenarios, and improves the generality and adaptability of the model. In addition, the present invention also provides a solid technical foundation for the popularization of speech synthesis technology in applications with multi-lingual, complex scenarios, and long speech inputs. Among them, the client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0066] Please refer to Figure 2 , Figure 2 shows a flowchart of an embodiment of the speech processing method according to the present invention. The speech processing method is applicable to a machine customer service in a financial scenario or an intelligent consultation robot in a medical scenario, and includes steps S100 to S500.

[0067] S100. Obtain the input speech feature sequence, and divide the speech feature sequence to obtain multiple adjacent feature blocks.

[0068] In this embodiment, the input speech feature sequence is divided into multiple blocks, which can better process long sequence data for better parallel computing, memory management, and feature extraction. Among them, the speech feature sequence can be divided by various methods, such as the fixed-size chunking method, the sliding window chunking method, the dynamic-size chunking method, and the chunking method based on time or other features. Among them, the fixed-size chunking method divides the sequence into multiple blocks according to a fixed size, the sliding window chunking method divides the sequence into multiple blocks using a sliding window, with overlap between each block, and the dynamic-size chunking method dynamically determines the size of each block according to certain conditions. For example, it can be chunked according to a specific threshold or pattern. If the sequence contains timestamps or other features, it can be chunked according to time or other features. In this embodiment, preferably, the method of chunking according to time is used to divide the input sequence into continuous non-overlapping fixed blocks. Exemplarily, if the input sequence has a time step of length 512 and the block size is 128, it is divided into 4 blocks (128×4 = 512). If the size of the last block after division is insufficient, the last block is padded.

[0069] Example illustration: In the medical field, the user conducts a consultation through an intelligent consultation robot and inputs voice information to the intelligent consultation robot. For example, the user inputs "I had a blood routine test today, and the result shows that the white blood cells are a bit high". After the system receives the voice information input by the user, it divides the voice information input by the user according to the timestamp, obtaining multiple feature blocks divided by the timestamp. For example, it is divided into feature blocks such as "I today", "had a blood routine test", "the result shows", "white blood cells", and "too high", and then each feature block is processed.

[0070] In the financial field, taking a bank intelligent customer service as an example, the user can input a problem represented by voice information through the intelligent customer service. For example, the user inputs "My bank card cannot be transferred". After the intelligent customer service system receives the voice information input by the user, it divides the voice information input by the user according to the timestamp, obtaining multiple feature blocks divided by the timestamp. For example, it is divided into feature blocks such as "My", "bank card", and "cannot be transferred", and then each feature block is processed.

[0071] S200. Perform context modeling on each of the feature blocks to extract the context information within each feature block.

[0072] In this embodiment, context modeling is performed inside each feature block to extract the meaning represented by each feature block. Here, context modeling refers to the process of modeling and representing the context relationships between elements in text or data through specific methods and models. In natural language processing, context usually refers to other relevant content around a certain word, sentence, or paragraph in the text, including the words before and after, sentence structure, semantic information, etc. For example, in the sentence "I want to consult about the illness", the context information of "illness" may include words such as "I want" and "consult", and these words together form a semantically related fragment to help understand the specific meaning of "illness" in this context.

[0073] S300. Based on the cross-block attention mechanism, establish the connection relationship between adjacent feature blocks.

[0074] In this embodiment, the cross-block attention mechanism allows the model to divide the data into multiple blocks when processing sequence or image data, and then perform attention calculation between different blocks, so as to capture the correlation information between different regions. The cross-block attention mechanism allows the model to divide the data into multiple blocks when processing sequence or image data, and then perform attention calculation between different blocks, so as to capture the correlation information between different regions. In tasks such as long text generation, document summarization, and machine translation, it can help the model better understand and generate text, improving the processing efficiency and effect. For example, when processing extremely long texts, the traditional self-attention mechanism has a high computational complexity, while the cross-block attention mechanism can effectively reduce the computational cost. Optionally, the embodiment of the present invention adopts the method of sparse attention or local attention to perform cross-block attention transfer.

[0075] S400. Encode the context information of each of the feature blocks, and group the encoded context information based on a preset clustering model to generate a discretized speech result.

[0076] In this embodiment, the speech features are encoded using encoding to extract high-level semantic information. On the basis of encoding, the clustering algorithm Deep Clustering is used to group the speech features to generate a more refined speech discretization result. During the training process, adversarial examples are introduced to simulate the changes in speech features under noise interference and complex scenarios. Through adversarial training and the deep learning framework, the model can better adapt to the requirements of speech tasks in complex scenarios, improving the robustness.

[0077] S500. Based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks, generate an output speech.

[0078] In this embodiment, the obtained discrete speech units are input into the large language model, which combines its own endogenous ability to perceive and generate speech with the discrete speech units containing rich features to generate corresponding speech output tokens.

[0079] In the embodiment of the present invention, first, an input speech feature sequence is obtained, and the speech feature sequence is divided to obtain a plurality of adjacent feature blocks; then, context modeling is performed on each of the feature blocks to extract context information within each feature block; then, based on the cross-block attention mechanism, a connection relationship is established between adjacent feature blocks; then, the context information of each of the feature blocks is encoded, and the encoded context information is grouped based on a preset clustering model to generate a discretized speech result; finally, based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks, an output speech is generated. By introducing a block attention mechanism and a more robust clustering algorithm, the present invention significantly improves the computational efficiency, robustness, and expressiveness of the model. The block attention mechanism significantly reduces the computational complexity and improves the long-context processing ability, and the more robust clustering algorithm enhances the accuracy and stability of speech discretization, improves the semantic retention effect, can better process multilingual speech data and complex noise scenarios, and improves the generality and adaptability of the model. In addition, the present invention also provides a solid technical foundation for the popularization of speech synthesis technology in applications such as multilingual, complex scenarios, and long speech input.

[0080] In some embodiments, please refer to Figure 3 , the step S100 specifically includes:

[0081] S110. Obtain an input speech feature sequence, and parse the time length corresponding to the speech feature sequence to obtain a time block corresponding to the speech feature sequence;

[0082] S120. Divide the time block corresponding to the speech feature sequence based on a preset size of the time division block;

[0083] S130. Divide the speech feature sequence based on the divided time blocks to obtain a plurality of adjacent feature blocks.

[0084] In this embodiment, the speech feature sequence is divided according to the size of the time length corresponding to the speech feature sequence. After the user inputs the speech feature sequence, the speech feature sequence corresponds to a time length. For example, when a patient inputs speech on the dialogue interface of a medical consultation robot, the time length is displayed on the dialogue interface, and this time length is the time block corresponding to the speech feature sequence. For example, the input sequence can be parsed into time steps with a length of 512. Then, based on the size of the preset time division block, this time block is divided. For example, if the size of the preset time division block is 128 time steps, the input sequence can be divided into 4 blocks. Then, according to the speech feature sequences corresponding to these four blocks, the overall speech feature sequence is divided, and multiple feature blocks are obtained, thereby reducing the computational complexity and improving the long context processing ability.

[0085] In some embodiments, step S120 specifically includes:

[0086] Determine whether the size of the time block corresponding to the speech feature sequence exceeds a preset value;

[0087] When the size of the time block corresponding to the speech feature sequence does not exceed the preset value, divide the time block corresponding to the speech feature sequence based on the size of the preset time division block;

[0088] When the size of the time block corresponding to the speech feature sequence exceeds the preset value, after increasing the size of the preset time division block, divide the time block corresponding to the speech feature sequence based on the increased size of the preset time division block;

[0089] Perform boundary adjustment and feature smoothing processing on the divided time blocks in sequence to obtain an optimized feature block sequence.

[0090] In this embodiment, since the sizes of the speech feature sequences input by the user are different, if a time division block of a fixed size is always used, the problem of an overly large single feature block will still occur. Therefore, the embodiments of the present invention can adaptively adjust the time division block. Exemplarily, the system presets a time block, for example, 1024 time steps. When the size of the time block corresponding to the speech feature sequence does not exceed the size of this time block, it indicates that the system can better process the speech feature sequence. At this time, the speech feature sequence can be divided according to the preset size of the time division block. When the size of the time block corresponding to the speech feature sequence input by the user exceeds the size of the time block preset by the system, the size of the preset time division block is adjusted. For example, if the size of the time block corresponding to the speech feature sequence input by the user is 2000 time steps and the original size of the time division block is 100 time steps, the size of the original time division block is increased at this time, for example, increased to 200 time steps. Furthermore, when processing long texts, the size of the block can be appropriately increased to reduce the number of blocks, thereby reducing the computational overhead.

[0091] In some embodiments, please refer to Figure 4 , the step S200 specifically includes:

[0092] S210. Calculate the first attention weights between each element in the feature block and other elements based on a preset self-attention mechanism;

[0093] S220. Perform weighted summation on each element in the feature block based on the calculated first attention weights to obtain the context information of each feature block;

[0094] S230. Perform feature compression, feature enhancement, and feature normalization processing on the context information in sequence to generate optimized feature block context information.

[0095] In this embodiment, a self-attention mechanism is used to implement context modeling. The self-attention mechanism allows elements in a single sequence to compare with each other and assigns a weight representing its relevance to other elements to each element. For example, when processing a sentence, a certain word can determine its importance based on its relationship with other words in the sentence. The self-attention mechanism can effectively capture the global dependencies in the sequence. No matter how far apart two elements are in the sequence, they can be directly connected. This is very effective for processing long sequences and complex language structures.

[0096] Specifically, for each element in the input sequence, the self-attention mechanism calculates the attention scores through a query vector, a key vector, and a value vector. Among them, the query vector is used to represent the focus of attention of the current element, the key vector is used to represent the features of the elements to be attended to, and the value vector contains the actual information of the elements to be attended to. During the calculation, first, the dot product similarity between the query vector and the key vector is calculated to obtain the attention scores between each pair of elements. Then, through a normalization function (such as the Softmax function), these scores are normalized to form a probability distribution to better represent the importance of each element. Finally, the attention scores are multiplied by the value vectors and summed to obtain the weighted representation of each element, that is, the context-aware representation. In this way, the representation of each element incorporates relevant information in the entire sequence, achieving context modeling.

[0097] In some embodiments, refer to Figure 5 , the step S300 specifically includes:

[0098] S310. Calculate the second attention weights between each feature block and other feature blocks based on a preset cross-block attention mechanism;

[0099] S320. Based on the calculated second attention weights, perform weighted summation on each of the feature blocks to obtain the connection relationship between adjacent feature blocks;

[0100] S330. Perform feature enhancement, feature normalization, and feature smoothing processing on the connection relationship in sequence to improve the stability and consistency of the connection relationship.

[0101] In this embodiment, the cross-block attention mechanism determines the correlation between different blocks by calculating the attention scores between them. This correlation can be based on content similarity, positional relationship, or other features. Then, according to these attention scores, the features of different blocks are weighted and combined to establish the connection between feature blocks. Among them, the weighted features are fused together to form a new feature representation. This new feature representation contains the information of different feature blocks and is appropriately weighted according to their correlation. In this way, the model can better utilize the information of different feature blocks and improve the ability to understand and process data.

[0102] In some embodiments, refer to Figure 6 , the step S400 specifically includes:

[0103] S410. Encode the context information of each of the feature blocks using a preset encoding model;

[0104] S420. Group the encoded context information based on a preset clustering model to generate a discretized speech result;

[0105] S430. Perform feature enhancement and feature normalization on the discretized speech result in sequence, and verify the speech result after the normalization process;

[0106] S440. Optimize and adjust the speech result after normalization according to the verification result to generate a discretized speech feature sequence.

[0107] In this embodiment, Wav2Vec is used for encoding. Among them, Wav2Vec encoding is based on a self-supervised learning framework. Through a multi-layer convolutional neural network (CNN), feature extraction is performed on the original audio signal, and the continuous audio waveform is converted into a low-dimensional latent representation. These latent representations can capture the semantic information and temporal information in the speech signal, can automatically learn representative features from a large amount of unlabeled speech data, and reduce the dependence on a large amount of labeled data. On the basis of encoding, the embodiment of the present invention also uses the clustering algorithm Deep Clustering to group the speech features to generate a finer speech discretization result. Among them, Deep Clustering is a model that uses deep learning technology to perform clustering analysis on speech signals. First, a neural network is trained to learn the embedding features of the speech signal. These embedding features can capture the semantic information and temporal information in the speech signal and provide a basis for subsequent clustering. After obtaining the embedding features, clustering algorithms (such as K-means, spectral clustering, etc.) are used to cluster these features. The purpose of clustering is to group similar speech signals into the same class, thereby realizing speech grouping.

[0108] In some embodiments, please refer to Figure 7 , the step S500 specifically includes:

[0109] S510. Input the discretized speech result into a preset large language model to generate a speech representation;

[0110] S520. Fuse the discretized speech result with the speech representation to generate a speech output result;

[0111] S530. Generate an output speech based on the connection relationship between adjacent feature blocks and the speech output result.

[0112] In this embodiment, the discrete speech units are feature-fused with the endogenous capabilities of the large language model (such as speech perception and generation capabilities). For example, the features of the discrete speech units are concatenated or weighted and summed with the speech representation of the large language model. Through the attention mechanism, the model can dynamically focus on the key features of the discrete speech units and, combined with the endogenous capabilities of the large language model, generate high-quality speech output tokens. During training, an end-to-end training method can be adopted, and the model can adaptively optimize the feature fusion and attention mechanism to generate higher-quality speech output. After obtaining the speech output result, a preset vocoder is used to synthesize speech from the speech output tokens to obtain the final output speech.

[0113] The technical solution provided by the present invention first obtains an input speech feature sequence, divides the speech feature sequence to obtain a plurality of adjacent feature blocks; then performs context modeling on each of the feature blocks to extract context information within each feature block; then based on a cross-block attention mechanism, establishes a connection relationship between adjacent feature blocks; then encodes the context information of each of the feature blocks, and groups the encoded context information based on a preset clustering model to generate a discretized speech result; finally, based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks, generates an output speech. The present invention significantly improves the computational efficiency, robustness, and expressiveness of the model by introducing a block attention mechanism and a more robust clustering algorithm. The block attention mechanism significantly reduces the computational complexity and improves the long context processing ability, and the more robust clustering algorithm enhances the accuracy and stability of speech discretization, improves the semantic retention effect, can better process multilingual speech data and complex noise scenarios, and improves the generality and adaptability of the model. In addition, the present invention also provides a solid technical foundation for the popularization of speech synthesis technology in applications such as multilingual, complex scenarios, and long speech input.

[0114] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0115] Another embodiment of the present invention provides a speech processing device, which corresponds one-to-one to the speech processing method in the above embodiment. Please refer to Figure 8 and this speech processing device includes a health portrait division module 11, a context information extraction module 12, a connection relationship establishment module 13, an encoding module 14, and an output module 15. The detailed description of each functional module is as follows:

[0116] The division module 11 is used to obtain an input speech feature sequence and divide the speech feature sequence to obtain a plurality of adjacent feature blocks.

[0117] The context information extraction module 12 is used to perform context modeling on each of the feature blocks to extract the context information within each feature block.

[0118] The connection relationship establishment module 13 is used to establish the connection relationship between adjacent feature blocks based on the cross-block attention mechanism.

[0119] The encoding module 14 is used to encode the context information of each of the feature blocks, and group the encoded context information based on a preset clustering model to generate a discretized speech result.

[0120] The output module 15 is used to generate the output speech based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks.

[0121] In some embodiments, the partitioning module 11 includes a time block acquisition unit, a time block partitioning unit, and a feature block partitioning unit, where

[0122] The time block acquisition unit is used to acquire the input speech feature sequence and parse out the time length corresponding to the speech feature sequence to obtain the time block corresponding to the speech feature sequence.

[0123] The time block partitioning unit is used to partition the time block corresponding to the speech feature sequence based on the preset size of the time partitioning block.

[0124] The feature block partitioning unit is used to partition the speech feature sequence based on the partitioned time block to obtain a plurality of adjacent feature blocks.

[0125] In some embodiments, the time block partitioning unit is specifically used for:

[0126] Determine whether the size of the time block corresponding to the speech feature sequence exceeds a preset value;

[0127] When the size of the time block corresponding to the speech feature sequence does not exceed the preset value, partition the time block corresponding to the speech feature sequence based on the preset size of the time partitioning block;

[0128] When the size of the time block corresponding to the speech feature sequence exceeds the preset value, after increasing the preset size of the time partitioning block, partition the time block corresponding to the speech feature sequence based on the increased preset size of the time partitioning block;

[0129] Perform boundary adjustment and feature smoothing processing on the partitioned time blocks in sequence to obtain an optimized feature block sequence.

[0130] In some embodiments, the context information extraction module 12 is specifically configured to:

[0131] Based on a preset self-attention mechanism, calculate the first attention weights between each element in the feature block and other elements;

[0132] Based on the calculated first attention weights, perform weighted summation on each element in the feature block to obtain the context information of each feature block;

[0133] Perform feature compression, feature enhancement, and feature normalization processing on the context information in sequence to generate optimized feature block context information.

[0134] In some embodiments, the connection relationship establishment module 13 is specifically configured to:

[0135] Based on a preset cross-block attention mechanism, calculate the second attention weights between each feature block and other feature blocks;

[0136] Based on the calculated second attention weights, perform weighted summation on each of the feature blocks to obtain the connection relationship between adjacent feature blocks;

[0137] Perform feature enhancement, feature normalization, and feature smoothing processing on the connection relationship in sequence to improve the stability and consistency of the connection relationship.

[0138] In some embodiments, the encoding module 14 is specifically configured to:

[0139] Adopt a preset encoding model to encode the context information of each feature block;

[0140] Based on a preset clustering model, group the encoded context information to generate a discretized speech result;

[0141] Perform feature enhancement and feature normalization processing on the discretized speech result in sequence, and verify the speech result after normalization processing;

[0142] According to the verification result, optimize and adjust the speech result after normalization to generate a discretized speech feature sequence.

[0143] In some embodiments, the output module 15 is specifically configured to:

[0144] Input the discretized speech result into a preset large language model to generate a speech representation;

[0145] Fuse the discretized speech result with the speech representation to generate a speech output result;

[0146] Generate an output speech based on the connection relationship between adjacent feature blocks and the speech output result.

[0147] In an embodiment of the present invention, first, an input speech feature sequence is obtained, and the speech feature sequence is partitioned to obtain a plurality of adjacent feature blocks; then, context modeling is performed on each of the feature blocks to extract context information within each feature block; thereafter, based on a cross-block attention mechanism, a connection relationship between adjacent feature blocks is established; then, the context information of each of the feature blocks is encoded, and the encoded context information is grouped based on a preset clustering model to generate a discretized speech result; finally, based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks, an output speech is generated. By introducing a block attention mechanism and a more robust clustering algorithm, the present invention significantly improves the computational efficiency, robustness, and expressiveness of the model. The block attention mechanism significantly reduces the computational complexity and improves the long-context processing ability. The more robust clustering algorithm enhances the accuracy and stability of speech discretization, improves the semantic retention effect, can better process multilingual speech data and complex noise scenarios, and improves the generality and adaptability of the model. In addition, the present invention also provides a solid technical foundation for the popularization of speech synthesis technology in applications such as multilingual, complex scenarios, and long speech input.

[0148] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0149] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0150] For the specific limitations of the voice processing device, reference may be made to the limitations on the voice processing method in the foregoing text, which will not be elaborated herein. Each module in the above voice processing device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0151] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 9 shown. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media, and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of an artificial intelligence-based voice processing method.

[0152] In one embodiment, a computer device is provided. The computer device can be a client, and its internal structure diagram can be as Figure 10 shown. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of an artificial intelligence-based voice processing method.

[0153] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0154] Obtain an input voice feature sequence, and divide the voice feature sequence to obtain a plurality of adjacent feature blocks;

[0155] Perform context modeling on each of the feature blocks to extract context information within each feature block;

[0156] Based on a cross-block attention mechanism, establish the connection relationship between adjacent feature blocks;

[0157] Encode the context information of each of the feature blocks, and group the encoded context information based on a preset clustering model to generate a discretized speech result;

[0158] Based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks, generate an output speech.

[0159] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0160] Obtain an input speech feature sequence, and divide the speech feature sequence to obtain a plurality of adjacent feature blocks;

[0161] Perform context modeling on each of the feature blocks to extract the context information within each feature block;

[0162] Based on a cross-block attention mechanism, establish the connection relationship between adjacent feature blocks;

[0163] Encode the context information of each of the feature blocks, and group the encoded context information based on a preset clustering model to generate a discretized speech result;

[0164] Based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks, generate an output speech.

[0165] It should be noted that for the functions or steps that can be achieved by the above computer-readable storage medium or computer device, reference can be made to the relevant descriptions on the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described in detail here.

[0166] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0168] In summary, the voice processing method, device, computer device, and storage medium provided by the present invention first obtain an input voice feature sequence, divide the voice feature sequence to obtain multiple adjacent feature blocks; then perform context modeling on each of the feature blocks to extract context information within each feature block; then establish a connection relationship between adjacent feature blocks based on a cross-block attention mechanism; then encode the context information of each of the feature blocks, group the encoded context information based on a preset clustering model to generate a discretized voice result; and finally generate an output voice based on a preset large voice model, the discretized voice result, and the connection relationship between adjacent feature blocks. By introducing a block attention mechanism and a more robust clustering algorithm, the present invention significantly improves the computational efficiency, robustness, and expressiveness of the model. The block attention mechanism significantly reduces the computational complexity and improves the long-context processing ability, and the more robust clustering algorithm enhances the accuracy and stability of voice discretization, improves the semantic retention effect, and can better process multilingual voice data and complex noise scenarios, enhancing the generality and adaptability of the model. In addition, the present invention also provides a solid technical foundation for the popularization of voice synthesis technology in applications such as multilingual, complex scenarios, and long voice input.

[0169] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for illustrative introduction and do not represent actual use.

[0170] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A voice processing method, characterized in that, The steps are as follows: Obtain the input speech feature sequence, and partition the speech feature sequence to obtain a plurality of adjacent feature blocks; Perform context modeling on each of the feature blocks to extract the context information within each feature block; Based on the cross-block attention mechanism, establish the connection relationship between adjacent feature blocks; Encode the context information of each of the feature blocks, and group the encoded context information based on a preset clustering model to generate a discretized speech result; Based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks, generate an output speech.

2. The voice processing method according to claim 1, wherein The obtaining the input speech feature sequence, and partitioning the speech feature sequence to obtain a plurality of adjacent feature blocks includes: Obtain the input speech feature sequence, and parse out the time length corresponding to the speech feature sequence to obtain the time block corresponding to the speech feature sequence; Based on the preset size of the time partition block, partition the time block corresponding to the speech feature sequence; Based on the partitioned time block, partition the speech feature sequence to obtain a plurality of adjacent feature blocks.

3. The voice processing method according to claim 2, wherein The partitioning the time block corresponding to the speech feature sequence based on the preset size of the time partition block includes: Judge whether the size of the time block corresponding to the speech feature sequence exceeds a preset value; When the size of the time block corresponding to the speech feature sequence does not exceed the preset value, partition the time block corresponding to the speech feature sequence based on the preset size of the time partition block; When the size of the time block corresponding to the speech feature sequence exceeds the preset value, after increasing the preset size of the time partition block, partition the time block corresponding to the speech feature sequence based on the increased preset size of the time partition block; Perform boundary adjustment and feature smoothing processing on the partitioned time blocks in sequence to obtain an optimized feature block sequence.

4. The voice processing method according to claim 1, wherein The performing context modeling on each of the feature blocks to extract the context information within each feature block includes: Based on the preset self-attention mechanism, calculate the first attention weights between each element in the feature block and other elements; Based on the calculated first attention weights, perform weighted summation on each element in the feature block to obtain the context information of each feature block; Perform feature compression, feature enhancement, and feature normalization processing on the context information in sequence to generate optimized feature block context information.

5. The voice processing method according to claim 1, characterized in that The establishing the connection relationship between adjacent feature blocks based on the cross-block attention mechanism includes: Based on the preset cross-block attention mechanism, calculate the second attention weights between each feature block and other feature blocks; Based on the calculated second attention weights, perform weighted summation on each of the feature blocks to obtain the connection relationship between adjacent feature blocks; Perform feature enhancement, feature normalization, and feature smoothing processing on the connection relationship in sequence to improve the stability and consistency of the connection relationship.

6. The voice processing method according to claim 1, wherein The encoding the context information of each of the feature blocks, and grouping the encoded context information based on a preset clustering model to generate a discretized speech result includes: Encode the context information of each of the feature blocks using a preset encoding model; Group the encoded context information based on a preset clustering model to generate a discretized speech result; Perform feature enhancement and feature normalization on the discretized speech result in sequence, and verify the speech result after the normalization process; Optimize and adjust the speech result after normalization according to the verification result to generate a discretized speech feature sequence.

7. The voice processing method according to claim 1, characterized in that Generating an output speech based on the preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks, includes: Input the discretized speech result into a preset large language model to generate a speech representation; Fuse the discretized speech result with the speech representation to generate a speech output result; Generate an output speech based on the connection relationship between adjacent feature blocks and the speech output result.

8. A voice processing device, characterized in that, Includes: A partitioning module, configured to obtain an input speech feature sequence, partition the speech feature sequence to obtain a plurality of adjacent feature blocks; A context information extraction module, configured to perform context modeling on each of the feature blocks to extract the context information within each feature block; A connection relationship establishment module, configured to establish a connection relationship between adjacent feature blocks based on a cross-block attention mechanism; An encoding module, configured to encode the context information of each of the feature blocks, and group the encoded context information based on a preset clustering model to generate a discretized speech result; An output module, configured to generate an output speech based on a preset large speech model, the discretized speech result, and the connection relationship between adjacent feature blocks.

9. A computer device, characterized in that, Includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the speech processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that, Computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the steps of the speech processing method according to any one of claims 1 to 7 are implemented.

Images