A method, device, electronic device and readable medium for generating reply content

Abstract

The embodiments of the present disclosure disclose a method, device, electronic device and computer-readable medium for generating reply content. A specific implementation of the method includes: obtaining user voice; determining at least one reply text corresponding to the user voice, and obtaining a reply text sequence; generating a voice vector according to the user voice; generating a text vector sequence according to pre-configured prompt information and a reply text sequence; generating reply content corresponding to the user voice according to the voice vector and the text vector sequence. This implementation improves the accuracy and efficiency of speech recognition and natural language understanding, provides users with a more convenient and intelligent voice interaction experience, and can shorten the process, speed up processing time, and achieve smoother interaction between machines and humans.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, electronic device, and computer-readable medium for generating reply content. Background Art

[0002] In the field of natural language processing, traditional speech recognition and text generation technologies usually require a combination of multiple modules, such as speech signal processing, feature extraction, acoustic model, language model, etc. These modules need to be manually designed and adjusted, and a large amount of manually annotated data is required to train the model. This approach has many problems, such as coupling between modules, data scarcity, and model complexity.

[0003] Most existing models are text input and text output. When humans have a conversation, they first convert the speech into text through speech recognition. The model outputs the answer to the conversation based on the text, and then plays it out through text-to-speech technology. This model is complex in structure and takes a long time to process. In addition, the emotional information of human expression is lost during the speech recognition process. The same text expressed in different tones has completely different meanings. Therefore, when communicating with humans, existing models are prone to problems such as being unable to correctly understand the content expressed by humans and slow to respond. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a reply content generation method, device, electronic device and computer-readable medium to solve the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a method for generating reply content, the method comprising: acquiring user voice; determining at least one reply text corresponding to the user voice to obtain a reply text sequence; generating a voice vector based on the user voice; generating a text vector sequence based on pre-configured prompt information and the reply text sequence; generating reply content corresponding to the user voice based on the voice vector and the text vector sequence.

[0007] In the second aspect, some embodiments of the present disclosure provide a reply content generation device, the device comprising: an acquisition unit, configured to acquire user voice; a determination unit, configured to determine at least one reply text corresponding to the above user voice, and obtain a reply text sequence; a first generation unit, configured to generate a voice vector according to the above user voice; a second generation unit, configured to generate a text vector sequence according to pre-configured prompt information and the above reply text sequence; a third generation unit, configured to generate reply content corresponding to the above user voice according to the above voice vector and the above text vector sequence.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which network device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation method in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0010] One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: the present disclosure improves the accuracy and efficiency of speech generation by automatically learning language rules and semantic information by using a large-scale corpus and a speech decoder. In addition, the speech decoder can directly generate text output from the speech signal without manually designing and adjusting multiple modules. In addition, the speech decoder can automatically learn language rules and semantic information, thereby improving the accuracy and efficiency of speech recognition and natural language understanding, and can more accurately recognize speech input, and can automatically learn and adapt to different speech input modes and language environments, thereby achieving more accurate and efficient speech recognition. At the same time, it can also realize a variety of speech processing functions such as speech synthesis and speech translation, providing users with a more convenient and intelligent speech interaction experience. In addition, the speech decoder improves the shortcomings of the previous model, directly inputs the speech input into the speech decoder and obtains the speech output, shortens the process, and speeds up the processing time. At the same time, the direct input audio stream completely retains the emotional information of humans when speaking, allowing the speech decoder to accurately understand human expressions and complete a smoother interaction between machines and humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1 is a schematic diagram of an application scenario of a method for generating reply content according to some embodiments of the present disclosure;

[0013] Figure 2 is a flowchart of some embodiments of the method for generating reply content according to the present disclosure;

[0014] Figure 3 is a schematic diagram of the structure of some embodiments of the reply content generating device according to the present disclosure;

[0015] Figure 4 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0018] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] Figure 1 It is a schematic diagram of an application scenario of the reply content generation method according to some embodiments of the present disclosure.

[0023] like Figure 1As shown, the server 101 can obtain the user voice 102, and then determine at least one reply text corresponding to the above user voice 102 to obtain a reply text sequence 103, and then generate a voice vector 104 based on the above user voice 102, and then generate a text vector sequence 106 based on the pre-configured prompt information 105 and the above reply text sequence 103, and finally, generate the reply content 107 corresponding to the above user voice 102 based on the above voice vector 104 and the above text vector sequence 106.

[0024] It is understandable that the reply content generation method can be executed by a terminal device, or it can also be executed by the server 101. The execution subject of the above method can also include a device formed by integrating the above terminal device and the above server 101 through a network, or it can also be executed by various software programs. Among them, the terminal device can be various electronic devices with information processing capabilities, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. The execution subject can also be embodied as a server 101, software, etc. When the execution subject is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0025] It should be understood that Figure 1 The number of servers in the example is only for illustration purposes. Any number of servers may be provided as required.

[0026] Continue to refer Figure 2 , shows a process 200 of some embodiments of the reply content generation method according to the present disclosure. The reply content generation method comprises the following steps:

[0027] Step 201, obtaining user voice.

[0028] In some embodiments, the execution subject of the reply content generation method (eg Figure 1 The server shown in the figure) can obtain the user voice through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0029] Specifically, after obtaining the user's voice, data preprocessing and voice denoising are usually performed:

[0030] 1. Data preprocessing: First, a large amount of speech data needs to be preprocessed, including sampling, filtering, feature extraction and other operations of speech signals, in order to facilitate subsequent speech recognition.

[0031] 2. Speech denoiser: Creating a speech denoiser can help eliminate noise in speech and improve the quality and clarity of speech signals. In practical applications, speech denoisers can be used in speech recognition, speech synthesis, speech communication and other fields to effectively improve the accuracy and reliability of speech processing. By analyzing and processing speech signals, the denoiser can identify and eliminate interference factors such as environmental noise, electronic noise, and speech distortion, thereby obtaining a clearer and more natural speech signal. In the process of implementing the speech denoiser, a series of signal processing algorithms and techniques (time domain filtering, frequency domain filtering, and wavelet transform) are used to achieve the best denoising effect.

[0032] Step 202, determining at least one reply text corresponding to the above user voice, and obtaining a reply text sequence.

[0033] In some embodiments, the execution entity may determine at least one reply text corresponding to the user voice to obtain a reply text sequence.

[0034] Step 203: Generate a speech vector according to the user speech.

[0035] In some embodiments, the execution entity may generate a speech vector based on the user speech.

[0036] In some optional implementations of some embodiments, the execution entity may input the user voice into a pre-trained voice encoder to obtain a voice vector, wherein a residual vector quantizer is provided in the voice encoder, and the residual vector quantizer is used to compress data.

[0037] Specifically, speech encoder: Creating a speech encoder can convert the denoised audio information into speech representations for processing and transmission in digital communications, speech recognition, speech synthesis and other fields. The main task of a speech encoder is to convert high-dimensional speech signals into low-dimensional speech representations for transmission and storage under limited bandwidth and storage space. In the process of implementing a speech encoder, a series of signal processing algorithms and techniques (linear predictive coding, vector quantization, wavelet transform) are used to achieve the best coding effect. Through a speech encoder, speech signals can be converted into digital signals, thereby realizing digital speech processing and transmission.

[0038] The main ones used are the previous speech encoder and the middle multi-layer residual vector quantizer (RVQ). The encoder converts the input audio x with a sampling rate of fs into an embedded sequence and outputs it to the quantizer.

[0039] Reason for using Residual Vector Quantizer (RVQ): The vectors generated by the speech encoder can take an infinite number of values. In order to transmit them to the receiver using a finite number of bits, they must be replaced with approximate vectors from a finite set codebook.

[0040] This approach works well for bitrates around 1kbps or less, but quickly reaches its limits when moving to higher bitrates. For example, even at a bitrate as low as 3kbps, assuming the encoder produces 100 vectors per second, a codebook of over 1 billion vectors needs to be stored, which is not feasible in practice. If a residual vector quantizer is used, the first layer quantizes the code vectors at a medium resolution, and each subsequent layer processes the residual of the previous layer. Splitting the quantization process into several layers can significantly reduce the codebook size. For example, at 3kbps and 100 vectors per second, using 5 quantization layers, the codebook size is reduced from 1 billion to 320.

[0041] The core idea of ​​RVQ is to use redundant information in data to improve the compression ratio while maintaining high coding efficiency, providing similar quality using much lower bandwidth.

[0042] The traditional linear prediction algorithm decomposes the speech signal into multiple linear components and a residual component. The advantage of this algorithm is that it has a relatively high compression rate, but it is very sensitive to noise. Therefore, during decomposition, compression and denoising require special processing optimization. The algorithm used for compression is similar to vector quantization. Traditional vector quantization cannot solve the problem of excessive codebook storage in the 3kbps scenario.

[0043] The present disclosure uses a residual vector quantizer to compress data, and each layer processes the residual of the previous layer to achieve a hierarchical progressive quantization process, thereby better removing the influence of noise.

[0044] Step 204, generating a text vector sequence according to the pre-configured prompt information and the above-mentioned reply text sequence.

[0045] In some embodiments, the execution entity may generate a text vector sequence according to pre-configured prompt information and the reply text sequence.

[0046] Specifically, the above-mentioned prompt refers to a serializer and deserializer that provides input text to the voice decoder to help the voice decoder better process input data. The present disclosure supports prompt input, and can customize the voice, emotion, etc. of the output voice of the voice decoder, thereby retaining the emotional information of humans when speaking.

[0047] Specifically, the execution subject can generate a text vector sequence through a text encoder. The text encoder uses the SpeechT5 model, which is also a Transformers model.

[0048] Text encoder: Creating a text encoder can convert the prompt information output by the user into a text representation, which is convenient for processing and analysis in natural language processing, machine translation, text classification and other fields. The main task of the text encoder is to convert high-dimensional text information into low-dimensional text representation, so that it can be stored and processed in a limited storage space. In the process of implementing the text encoder, a series of natural language processing algorithms and technologies (word embedding, convolutional neural network, recurrent neural network) are used to achieve the best encoding effect. Through the text encoder, text information can be converted into a vector representation, thereby realizing the digital processing and analysis of text information.

[0049] Step 205: Generate a reply content corresponding to the user's voice based on the voice vector and the text vector sequence.

[0050] In some embodiments, the execution entity may generate a reply content corresponding to the user voice based on the voice vector and the text vector sequence.

[0051] In some optional implementation methods of some embodiments, the above-mentioned execution entity can input the above-mentioned text vector sequence and the above-mentioned voice vector into a pre-trained voice decoder to obtain a reply vector corresponding to the above-mentioned user voice; generate a reply voice and / or reply text according to the above-mentioned reply vector; and use the above-mentioned reply voice and / or the above-mentioned reply text as the reply content.

[0052] As an example, the user voice may be "What's the weather in Shanghai today?", and the prompt information may be a model prompt information such as "Answer with a gentle female voice". This voice and text will be parsed into vectors with feature information (text vector and voice vector) by two encoders (text encoder and voice encoder), and then input into the voice decoder. The voice decoder then outputs a reply vector and generates a gentle female voice as the reply content based on the reply vector, such as a gentle female voice "The weather in Shanghai is currently sunny."

[0053] In some optional implementations of some embodiments, the above-mentioned speech decoder is composed of an input layer, at least two convolutional layers, at least two pooling layers, multiple activation layers and a fully connected layer, the above-mentioned convolutional layer includes a first convolutional layer composed of five convolutional kernels and a second convolutional layer composed of ten convolutional kernels, and the above-mentioned first convolutional layer and the second convolutional layer are connected to the pooling layer and the activation layer in sequence.

[0054] In some optional implementations of some embodiments, the above-mentioned speech decoder is trained according to the following steps: obtaining a training sample set, wherein the above-mentioned training sample set includes a sample speech vector and a sample text vector sequence, and a sample reply vector corresponding to the above-mentioned sample speech vector and the sample text vector sequence; inputting the above-mentioned sample speech vector and the sample text vector sequence into the model to be trained to obtain a reply vector; comparing the above-mentioned reply vector with the above-mentioned sample reply vector to obtain a comparison result; in response to determining that the above-mentioned comparison result meets a preset condition, using the above-mentioned model to be trained as a speech decoder.

[0055] Specifically, the sample speech vector can be obtained by inputting the speech of a person into a speech encoder, and the sample text vector is usually generated based on the input sample prompt. For example, the content of the sample speech vector is "What is the weather in Shanghai today?", and the sample prompt information may be model prompt information such as "Answer with a gentle female voice", and the sample reply vector corresponding to this sample speech vector and sample text vector may be the voice of a female with a gentle voice (expected output), such as "Shanghai is currently sunny", and this sample speech vector and sample text vector will be parsed by two encoders into a reply vector (actual output) with feature information, and the actual output will be compared with the expected output, and the speech decoder will be optimized according to the comparison results, thereby completing training again and again.

[0056] In some optional implementations of some embodiments, the execution entity may optimize the model to be trained by alternately performing forward propagation and back propagation in response to determining that the comparison result does not satisfy a preset condition.

[0057] In some optional implementations of some embodiments, the execution subject may use forward propagation to calculate the output result through the training data and weight parameters; use back propagation to calculate the gradient of the loss function to each parameter through the derivative chain rule, use the gradient to simulate the momentum to replace the gradient, and optimize the model to be trained according to the following formula: Where γ represents the supermomentum parameter, which satisfies 0≤γ<1, Vt represents the velocity at time t, η t represents the learning rate and t represents the time step.

[0058] Compared with the traditional method, the independent variable of the present disclosure at each time step (t) is approximately the exponentially weighted moving average of the update amount corresponding to the most recent 1 / (1-γ) time steps (t) of the former and then divided by 1-γ. Therefore, the movement amplitude of the independent variable in each direction in the momentum method depends not only on the current gradient, but also on whether the past gradients are consistent in each direction. This method can make it easier to find the optimal solution, and when the gradient is not high, it will descend faster.

[0059] Specifically, during the training process of the neural network, forward propagation and back propagation are performed alternately. Forward propagation calculates the output result through training data and weight parameters; back propagation calculates the gradient of the loss function for each parameter through the chain rule of derivatives, and updates the parameters according to the gradient.

[0060] Forward propagation process formula: Among them, z (l) represents the input vector of the lth layer, ω (l) represents the weight matrix from layer l-1 to layer l, a (l) represents the activation output vector of the lth layer. b represents the bias vector, σ(z (l) ) is the activation function, l = 2, 3, 4...L (L represents the last layer), the output of each layer will become the input of the next layer, and finally a function set will be obtained.

[0061] Back propagation process formula: δ (l) =((ω (l+1) ) T δ (l+1) )⊙σ′(z (l) ), δ (l) Represents the loss vector of the loss function at layer l, ⊙ is represented by the product of the corresponding elements of two matrices, l = L-1, L-2, L-3...2.

[0062] Specifically, speech decoder: The speech decoder can input speech representation and text representation into the decoder to obtain speech output. The main task of the speech decoder is to convert low-dimensional speech representation and text representation into high-dimensional speech signals for processing and generation in the fields of speech synthesis and speech conversion. In the process of implementing the speech decoder, a series of signal processing algorithms and technologies (deconvolutional neural network, recurrent neural network) are used to achieve the best decoding effect. Through the decoder, speech representation and text representation can be converted into speech signals, thereby realizing applications such as speech synthesis and speech conversion.

[0063] The neural network structure used by the speech decoder is similar to CNNLM, which consists of an input layer, at least two convolutional layers, at least two pooling layers, multiple activation layers, and a fully connected layer:

[0064] Input layer: Perform preliminary processing on the input raw data, first remove the mean of the data, concentrate the data near the origin, and then normalize it to integrate the data features of each dimension into the same dimension. Finally, use the WordEmbedding algorithm to map the data into groups of vectors.

[0065] Convolution layer: The input data is convolved with a set of learnable convolution kernels, and a certain amount of padding is added around the edges of the input data to keep the size of the output and input data unchanged during the convolution operation. When the stride of the convolution operation is set to a unit length, the relationship between the filter size and the number of padding layers is as follows: P = (F-1) / 2, where p is the number of padding layers to the outside, and F is the filter size. The convolution operation can be regarded as a filtering operation. The values ​​obtained by each filter through matrix operations are combined to obtain a new output matrix. In short, the role of the convolution layer is to extract various features of the input data.

[0066] Activation layer: Activate the data of the convolution layer through the ReLU function to enhance the recognition ability of the model.

[0067] Pooling layer: The pooling layer is usually in the middle of multiple convolutional layers. Its function is to compress data, reduce overfitting, and enhance the characteristic information of the sound. If a piece of music is partially distorted, you can still hear what kind of music it is.

[0068] Fully connected layer: Usually at the end of the neural network, it connects the layers through different weights. The output of the pooling layer is connected to one or more fully connected layers for feature fusion and classification.

[0069] Training algorithm and optimization:

[0070] The algorithm used for training the model is BP. The basic idea is to calculate the gradient of the loss function to the network parameters through the chain rule. Forward propagation starts from the input layer and calculates the output value of each layer in turn according to the network structure until the final output value of the network is calculated. During the forward propagation process, the input and output values ​​of each layer need to be saved for subsequent calculation of the gradient. Backward propagation means starting from the output layer, calculating the gradient value of each layer in turn according to the network structure until the gradient of the network parameters is calculated. During the back propagation process, the gradient of each layer needs to be calculated according to the chain rule, and the network parameters are updated using optimization algorithms such as gradient descent. In short, the main body of the back propagation algorithm is to calculate the gradients of the output layer and the hidden layer, so as to calculate and optimize the weight matrix.

[0071] The optimization algorithm refers to the SGD algorithm. The gradient is used to simulate the momentum data instead of the gradient: The super momentum parameter γ satisfies 0≤γ<1. The advantage of this optimization algorithm is that it can find the optimal solution better than the conventional SGD algorithm. If the descent angle is small, the momentum will decrease faster.

[0072] Training samples:

[0073] Audio data samples are usually represented in the form of time-frequency graphs, and the frequency and intensity at each moment are usually converted into the form of pixel matrices. Training samples usually include audio data and corresponding labels or classifications.

[0074] In addition, the present disclosure can also realize the functions of speech recognition, speech synthesis and speech translation:

[0075] Speech recognition: In the process of speech recognition, the input speech signal is converted into text output and then input into the speech decoder. Through the reasoning process of the speech decoder, the corresponding text output is obtained. In this process, factors such as noise, accent, and speaking speed of the speech signal need to be considered to improve the accuracy and robustness of recognition.

[0076] Speech synthesis: In the process of speech synthesis, text needs to be input into the speech decoder, and the corresponding speech output is obtained through the inference process of the speech decoder. In this process, factors such as the fluency and naturalness of the speech need to be considered to improve the quality and readability of the synthesized speech.

[0077] Speech translation: In the process of speech translation, the input speech signal needs to be converted into text output in the target language. The specific implementation method includes inputting the speech signal into the speech decoder, obtaining the corresponding source language text output through the model's reasoning process, and then inputting the source language text into the translation model, and obtaining the corresponding target language text output through the model's reasoning process. In this process, the correspondence between the speech signal and the text needs to be considered to improve the accuracy and fluency of the translation.

[0078] In summary, the specific implementation of the speech decoder involves multiple aspects, including data preprocessing, model training, speech recognition, speech synthesis and speech translation, etc. Through the application of these technical means, a more accurate and efficient voice interaction experience can be achieved, providing users with more convenient and intelligent voice services.

[0079] One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: the present disclosure improves the accuracy and efficiency of speech generation by automatically learning language rules and semantic information by using a large-scale corpus and a speech decoder. In addition, the speech decoder can directly generate text output from the speech signal without manually designing and adjusting multiple modules. In addition, the speech decoder can automatically learn language rules and semantic information, thereby improving the accuracy and efficiency of speech recognition and natural language understanding, and can more accurately recognize speech input, and can automatically learn and adapt to different speech input modes and language environments, thereby achieving more accurate and efficient speech recognition. At the same time, it can also realize a variety of speech processing functions such as speech synthesis and speech translation, providing users with a more convenient and intelligent speech interaction experience. In addition, the speech decoder improves the shortcomings of the previous model, directly inputs the speech input into the speech decoder and obtains the speech output, shortens the process, and speeds up the processing time. At the same time, the direct input audio stream completely retains the emotional information of humans when speaking, allowing the speech decoder to accurately understand human expressions and complete a smoother interaction between machines and humans.

[0080] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a reply content generation device, and these device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0081] like Figure 3 As shown, the reply content generating device 300 of some embodiments includes: an acquisition unit 301, a determination unit 302, a first generation unit 303, a second generation unit 304 and a third generation unit 305. Among them, the acquisition unit 301 is configured to acquire user speech; the determination unit 302 is configured to determine at least one reply text corresponding to the above user speech to obtain a reply text sequence; the first generation unit 303 is configured to generate a speech vector according to the above user speech; the second generation unit 304 is configured to generate a text vector sequence according to the pre-configured prompt information and the above reply text sequence; and the third generation unit 305 is configured to generate the reply content corresponding to the above user speech according to the above speech vector and the above text vector sequence.

[0082] In an optional implementation of some embodiments, the third generation unit 305 is further configured to: input the above-mentioned text vector sequence and the above-mentioned voice vector into a pre-trained voice decoder to obtain a reply vector corresponding to the above-mentioned user voice; generate a reply voice and / or a reply text according to the above-mentioned reply vector; and use the above-mentioned reply voice and / or the above-mentioned reply text as the reply content.

[0083] In an optional implementation of some embodiments, the above-mentioned speech decoder is composed of an input layer, at least two convolutional layers, at least two pooling layers, multiple activation layers and a fully connected layer, the above-mentioned convolutional layer includes a first convolutional layer composed of five convolutional kernels and a second convolutional layer composed of ten convolutional kernels, and the above-mentioned first convolutional layer and the second convolutional layer are connected to the pooling layer and the activation layer in sequence.

[0084] In an optional implementation of some embodiments, the speech decoder is trained according to the following steps: obtaining a training sample set, wherein the training sample set includes a sample speech vector and a sample text vector sequence, and a sample reply vector corresponding to the sample speech vector and the sample text vector sequence; inputting the sample speech vector and the sample text vector sequence into the model to be trained to obtain a reply vector; comparing the reply vector with the sample reply vector to obtain a comparison result; in response to determining that the comparison result meets a preset condition, using the model to be trained as a speech decoder.

[0085] In an optional implementation of some embodiments, the above-mentioned device also includes an optimization unit, which is configured to: in response to determining that the above-mentioned comparison result does not meet the preset conditions, optimize the above-mentioned model to be trained by alternately using forward propagation and back propagation.

[0086] In an optional implementation of some embodiments, the optimization unit is further configured to use forward propagation to calculate the output result through the training data and weight parameters; use back propagation to calculate the gradient of the loss function to each parameter through the derivative chain rule, use the gradient to simulate the momentum to replace the gradient, and optimize the above-mentioned model to be trained according to the following formula: Where γ represents the supermomentum parameter, which satisfies 0≤γ<1, Vt represents the velocity at time t, η t represents the learning rate and t represents the time step.

[0087] In an optional implementation of some embodiments, the first generation unit 303 is further configured to input the above-mentioned user voice into a pre-trained voice encoder to obtain a voice vector, wherein a residual vector quantizer is provided in the above-mentioned voice encoder, and the above-mentioned residual vector quantizer is used to compress data.

[0088] It is understood that the units described in the device 300 are similar to those described in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 300 and the units contained therein, and will not be described in detail here.

[0089] One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: the present disclosure improves the accuracy and efficiency of speech generation by automatically learning language rules and semantic information by using a large-scale corpus and a speech decoder. In addition, the speech decoder can directly generate text output from the speech signal without manually designing and adjusting multiple modules. In addition, the speech decoder can automatically learn language rules and semantic information, thereby improving the accuracy and efficiency of speech recognition and natural language understanding, and can more accurately recognize speech input, and can automatically learn and adapt to different speech input modes and language environments, thereby achieving more accurate and efficient speech recognition. At the same time, it can also realize a variety of speech processing functions such as speech synthesis and speech translation, providing users with a more convenient and intelligent speech interaction experience. In addition, the speech decoder improves the shortcomings of the previous model, directly inputs the speech input into the speech decoder and obtains the speech output, shortens the process, and speeds up the processing time. At the same time, the direct input audio stream completely retains the emotional information of humans when speaking, allowing the speech decoder to accurately understand human expressions and complete a smoother interaction between machines and humans.

[0090] Reference below Figure 4 , which shows an electronic device (eg, Figure 1 A schematic diagram of the structure of the server 400. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0091] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0092] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0093] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0094] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0095] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0096] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the user's voice; determines at least one reply text corresponding to the user's voice to obtain a reply text sequence; generates a voice vector according to the user's voice; generates a text vector sequence according to the pre-configured prompt information and the reply text sequence; generates the reply content corresponding to the user's voice according to the voice vector and the text vector sequence.

[0097] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor includes an acquisition unit, a determination unit, a first generation unit, a second generation unit, and a third generation unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring user voice".

[0100] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0101] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for generating reply content, comprising: Get user voice; Determine at least one reply text corresponding to the user voice, and obtain a reply text sequence; Generate a speech vector according to the user speech; Generate a text vector sequence according to the preconfigured prompt information and the reply text sequence; Generate a reply content corresponding to the user voice according to the voice vector and the text vector sequence; The step of generating a reply content corresponding to the user voice according to the voice vector and the text vector sequence includes: Inputting the text vector sequence and the speech vector into a pre-trained speech decoder to obtain a response vector corresponding to the user speech; Generate a reply voice and / or a reply text according to the reply vector; Using the reply voice and / or the reply text as reply content; The speech decoder is composed of an input layer, at least two convolutional layers, at least two pooling layers, multiple activation layers and a fully connected layer, wherein the convolutional layer includes a first convolutional layer composed of five convolutional kernels and a second convolutional layer composed of ten convolutional kernels, and the first convolutional layer and the second convolutional layer are sequentially connected to the pooling layer and the activation layer; The speech decoder is trained according to the following steps: Acquire a training sample set, wherein the training sample set includes a sample speech vector and a sample text vector sequence, and a sample answer vector corresponding to the sample speech vector and the sample text vector sequence; Inputting the sample speech vector and the sample text vector sequence into the model to be trained to obtain a response vector; Compare the reply vector with the sample reply vector to obtain a comparison result; In response to determining that the comparison result satisfies a preset condition, using the model to be trained as a speech decoder; The method further comprises: In response to determining that the comparison result does not meet the preset condition, optimizing the model to be trained by alternately performing forward propagation and back propagation; The method of optimizing the model to be trained by alternately performing forward propagation and back propagation includes: Use forward propagation to calculate the output results through training data and weight parameters; Back propagation is used to calculate the gradient of the loss function to each parameter through the derivative chain rule, and the gradient is used to simulate the momentum to replace the gradient, and the model to be trained is optimized according to the following formula: Wherein, γ represents the supermomentum parameter, and the supermomentum parameter satisfies 0≤γ<1, V t represents the speed at time t, η t represents the learning rate, t represents the time step; The generating a speech vector according to the user speech comprises: The user voice is input into a pre-trained speech encoder to obtain a speech vector, wherein the speech encoder is provided with a residual vector quantizer, and the residual vector quantizer is used to compress data.

2. A reply content generating device, comprising: An acquisition unit, configured to acquire user voice; a determining unit, configured to determine at least one reply text corresponding to the user voice, and obtain a reply text sequence; A first generating unit, configured to generate a speech vector according to the user speech; A second generating unit, configured to generate a text vector sequence according to pre-configured prompt information and the reply text sequence; A third generating unit is configured to generate a reply content corresponding to the user voice according to the voice vector and the text vector sequence; The third generating unit is specifically used for: Input the text vector sequence and the speech vector into a pre-trained speech decoder to obtain a reply vector corresponding to the user speech; generate a reply speech and / or a reply text according to the reply vector; and use the reply speech and / or the reply text as the reply content; The speech decoder is composed of an input layer, at least two convolutional layers, at least two pooling layers, multiple activation layers and a fully connected layer, wherein the convolutional layer includes a first convolutional layer composed of five convolutional kernels and a second convolutional layer composed of ten convolutional kernels, and the first convolutional layer and the second convolutional layer are sequentially connected to the pooling layer and the activation layer; The speech decoder is trained according to the following steps: Acquire a training sample set, wherein the training sample set includes a sample speech vector and a sample text vector sequence, and a sample reply vector corresponding to the sample speech vector and the sample text vector sequence; input the sample speech vector and the sample text vector sequence into a model to be trained to obtain a reply vector; compare the reply vector with the sample reply vector to obtain a comparison result; in response to determining that the comparison result meets a preset condition, use the model to be trained as a speech decoder; The third generating unit is further used for: In response to determining that the comparison result does not meet the preset condition, optimizing the model to be trained by alternately performing forward propagation and back propagation; The third generating unit is specifically used for: Use forward propagation to calculate the output results through training data and weight parameters; Back propagation is used to calculate the gradient of the loss function to each parameter through the derivative chain rule, and the gradient is used to simulate the momentum to replace the gradient, and the model to be trained is optimized according to the following formula: Wherein, γ represents the supermomentum parameter, and the supermomentum parameter satisfies 0≤γ<1, V t represents the speed at time t, η t represents the learning rate, t represents the time step; The first generating unit is specifically used for: The user voice is input into a pre-trained speech encoder to obtain a speech vector, wherein the speech encoder is provided with a residual vector quantizer, and the residual vector quantizer is used to compress data.

3. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in claim 1.

4. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to claim 1 is implemented.

Images