A voice dialog method, system, device and medium

Abstract

The application discloses a voice dialogue method, system, device and medium, wherein the method inputs a first target voice signal into an audio coding network for signal coding to obtain a first audio coding unit sequence, the first audio coding unit sequence includes a plurality of audio coding units, and each audio coding unit corresponds to an audio frame in the first target voice signal; inputs the first target voice signal and the first audio coding unit sequence into a large language model for disambiguation reasoning to obtain an audio response unit sequence output by the large language model; inputs the audio response unit sequence into an audio decoding network for signal waveform reconstruction to obtain a voice response signal, and performs dialogue response on the first target voice signal of the user according to the voice response signal. The method can effectively improve the response speed of voice dialogue and is beneficial to improving user experience. The application relates to the technical field of voice recognition.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech dialogue method, system, device and medium. Background Technology

[0002] As artificial intelligence technology is increasingly applied in the field of education, voice dialogue with users is gradually becoming an important auxiliary function in educational companionship scenarios.

[0003] Currently, the relevant technologies typically involve first converting the user's voice signal into speech-to-text, then using a large language model to generate response text based on the speech-to-text, and finally synthesizing speech output based on the response text to achieve voice dialogue with the user. This method has high response latency (usually higher than 2 seconds), the dialogue structure with the user is often not natural, and the user experience is poor.

[0004] Therefore, the problems existing in the current technology still need to be solved and optimized. Summary of the Invention

[0005] To address at least one of the aforementioned technical problems, this application provides a voice dialogue method, system, device, and medium, wherein the method can effectively improve the response speed of voice dialogue and is beneficial to improving user experience.

[0006] According to a first aspect of this application, a voice dialogue method is provided, comprising: The trained audio codec network and the user's first target speech signal are acquired; the trained audio codec network includes an audio encoding network and an audio decoding network; The first target speech signal is input into the audio coding network for signal encoding to obtain a first audio coding unit sequence. The first audio coding unit sequence includes a plurality of audio coding units, and each audio coding unit corresponds to an audio frame in the first target speech signal. The first target speech signal and the first audio coding unit sequence are input into a large language model for disambiguation inference to obtain the audio response unit sequence output by the large language model. The audio response unit sequence is input into the audio decoding network for signal waveform reconstruction to obtain a speech response signal, and a dialogue response is performed on the user's first target speech signal based on the speech response signal.

[0007] In some embodiments, the audio encoding network includes an encoding module and a quantization module, and the trained audio codec network is obtained through the following steps: Obtain a first speech training group and a second speech training group; the first speech training group is a speech training group without audio labels; the second speech training group is a speech training group with audio labels. Freeze the network parameters of the encoding module, and pre-train the quantization module and audio decoding network in the audio codec network according to the first speech training group to obtain the pre-trained quantization module and audio decoding network; Unfreeze the network parameters of the encoding module, and fine-tune the encoding module, the trained quantization module, and the audio decoding network according to the second speech training group to obtain the trained audio codec network.

[0008] In some embodiments, inputting the first target speech signal into the audio coding network for signal encoding to obtain a first audio coding unit sequence includes: Contextual features are extracted from the first target speech signal to obtain contextual embedding features; The context embedding features are refined using multidimensional features to obtain several audio coding units; Hierarchical residual vector quantization is performed on all the audio coding units to obtain the first audio coding unit sequence.

[0009] In some embodiments, the multidimensional feature refinement of the context embedding features to obtain a plurality of audio coding units includes: Perform tone analysis on the context embedding features to obtain intermediate tone features; Prosodic analysis is performed on the context embedding features to obtain intermediate prosodic features; The context embedding features are semantically refined to obtain intermediate semantic features; The audio coding unit is obtained based on the intermediate tone feature, the intermediate prosodic feature, and the intermediate semantic feature.

[0010] In some embodiments, the step of inputting the first target speech signal and the first audio coding unit sequence into a large language model for disambiguation inference to obtain the audio response unit sequence output by the large language model includes: Obtain the original text unit sequence corresponding to the first target speech signal; Based on the first audio coding unit sequence, semantic disambiguation is performed on the original text unit sequence to obtain a text disambiguation unit sequence; The text disambiguation unit sequence is subjected to an audio response to obtain the audio response unit sequence.

[0011] In some embodiments, the step of semantically disambiguating the original text unit sequence based on the first audio coding unit sequence to obtain a text disambiguation unit sequence includes: For the first audio coding unit sequence, the original text unit sequence is spliced ​​and projected to obtain multimodal embedding features; The multimodal embedding features are subjected to feature fusion and disambiguation processing to obtain the text disambiguation unit sequence.

[0012] In some embodiments, the method further includes: Obtain the audio mutation threshold and the second audio coding unit sequence output by the audio coding network in the dialogue response state; The second audio coding unit sequence is reconstructed to obtain the reconstructed signal; Based on the audio mutation threshold, signal mutation detection is performed on the reconstructed signal to obtain mutation detection results; If the mutation detection result indicates the presence of a mutation, then a dialogue response is performed on the second target speech signal corresponding to the second audio coding unit sequence.

[0013] According to a second aspect of this application, a voice dialogue system is provided, comprising: The first processing unit is used to acquire a trained audio codec network and the user's first target speech signal; the trained audio codec network includes an audio encoding network and an audio decoding network. The second processing unit is used to input the first target speech signal into the audio coding network for signal encoding to obtain a first audio coding unit sequence. The first audio coding unit sequence includes a plurality of audio coding units, and each audio coding unit corresponds to an audio frame in the first target speech signal. The third processing unit is used to input the first target speech signal and the first audio coding unit sequence into the large language model for disambiguation inference, and obtain the audio response unit sequence output by the large language model. The fourth processing unit is used to input the audio response unit sequence into the audio decoding network for signal waveform reconstruction to obtain a voice response signal, and to perform a dialogue response to the user's first target voice signal based on the voice response signal.

[0014] According to a third aspect of this application, an electronic device is provided, comprising: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described above.

[0015] According to a fourth aspect of this application, a computer-readable storage medium is provided, wherein a processor-executable program is stored, the processor-executable program being used, when executed by the processor, to implement the method as described above.

[0016] According to a fifth aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the method described above.

[0017] The beneficial effects of the technical solutions provided in this application are: This application provides a voice dialogue method, system, device, and medium. The method acquires a trained audio codec network and a user's first target speech signal. The trained audio codec network includes an audio encoding network and an audio decoding network. The first target speech signal is input into the audio encoding network for signal encoding to obtain a first audio encoding unit sequence, which includes several audio encoding units, each corresponding to an audio frame in the first target speech signal. The first target speech signal and the first audio encoding unit sequence are input into a large language model for disambiguation inference to obtain an audio response unit sequence output by the large language model. The audio response unit sequence is input into the audio decoding network for signal waveform reconstruction to obtain a speech response signal. Based on the speech response signal, a dialogue response is performed on the user's first target speech signal. This method uses a large language model to perform disambiguation inference on the first target speech signal and the first audio encoding unit sequence, which can directly generate an audio response unit sequence to achieve subsequent dialogue responses. This can effectively improve the response speed of voice dialogue and enhance the user experience. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a voice dialogue method provided in an embodiment of this application; Figure 2 A schematic diagram of the training process of an audio codec network provided in an embodiment of this application; Figure 3 A detailed flowchart of step S120 provided for an embodiment of this application; Figure 4 A detailed flowchart of step S320 provided for an embodiment of this application; Figure 5 A schematic diagram of a training process for step S130 provided in an embodiment of this application; Figure 6A detailed flowchart of step S520 provided for an embodiment of this application; Figure 7 A schematic diagram of one optional process of a voice dialogue method provided in an embodiment of this application; Figure 8 A schematic diagram of the framework of a voice dialogue system provided in an embodiment of this application; Figure 9 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] The present application will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0020] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Currently, the relevant technologies typically involve first converting the user's voice signal into speech-to-text, then using a large language model to generate response text based on the speech-to-text, and finally synthesizing speech output based on the response text to achieve voice dialogue with the user. This approach has the following drawbacks in educational companionship scenarios: 1. High response latency (usually higher than 2 seconds), the dialogue structure with users is often not natural enough, it is difficult to maintain the attention of some users, and the user experience is poor.

[0023] 2. Loss of emotional information: Key non-linguistic features such as intonation, pauses, and emotions in the speech signal are stripped away during the speech-to-text conversion stage, making subsequent dialogue responses mechanical, monotonous, and lacking in emotion, resulting in a poor user experience.

[0024] 3. Poor adaptability to speech acoustic characteristics. Since Chinese is a tonal language, there are objective factors such as the same syllable with different tones corresponding to completely different semantic meanings (such as "妈" and "骂"), as well as a relatively large number of polyphonic characters and homophones. In cases where the user's pronunciation is blurred, the speech rate is unstable, or the words used are unconventional, the recognition rate of converting speech signals into speech texts significantly decreases, the semantic ambiguity rate is high, and the incorrect text further causes misunderstandings in the subsequent large language model (LLM), forming an error propagation chain, and the quality of the dialogue response is unsatisfactory.

[0025] 4. Unable to support unstructured interaction behaviors. Because the system depends on complete speech input to start processing, it is difficult to achieve a dialogue response to the user's latest speech signal in cases where the user interrupts midway.

[0026] It should be noted that the related technologies mentioned above are only used to assist in understanding the technical solution of this application, and it does not mean that they belong to the publicly disclosed prior art.

[0027] In view of this, the embodiments of this application provide a speech dialogue method, system, device, and medium. Among them, this method decodes the information of the target speech signal, specifically through multi-dimensional feature refinement and hierarchical residual vector quantization processing. It can explicitly model the association between Chinese tones and semantics, significantly reduce semantic errors caused by polyphonic characters and homophones, and improve the robustness to the user's blurred pronunciation, thereby being beneficial to improving the quality of subsequent dialogue responses.

[0028] Moreover, this method performs disambiguation reasoning on the first target speech signal and the first audio coding unit sequence through the large language model. Specifically, it performs semantic disambiguation on the original text unit sequence through the audio coding unit sequence and generates the corresponding audio response unit sequence. It can省去 the steps of converting the speech signal into a speech text and synthesizing the answer text into speech output, effectively reducing the dialogue response delay, and thus effectively improving the user experience. Also, it also applies the semantic, tonal, prosodic and other feature information in the audio coding unit sequence to the original text unit sequence for semantic disambiguation. It can保留 the key non-verbal features in the speech signal, making the subsequent dialogue response have relatively rich emotions, which is beneficial to improving the user experience.

[0029] In addition, this method also performs frame ability mutation detection, which can perform a dialogue response to the user's latest speech signal in cases where the user interrupts midway during the dialogue response process.

[0030] A speech dialogue method provided by the embodiments of this application can be specifically described through the following embodiments. First, a speech dialogue method in the embodiments of this application is described.

[0031] It should be noted that in the translation of , the Chinese word "省去" is translated as "省去", which is a literal translation. In English, a more appropriate expression might be "omit" or "eliminate". Also, the Chinese word "保留" is translated as "保留", and a more common English expression could be "retain" or "preserve". These are just suggestions for improvement, and the translation follows the requirement of preserving the original text as closely as possible.The voice dialogue method provided in this application can be applied to educational companionship application scenarios. In these scenarios, the educational companionship service provider can respond to the user's voice signals through a terminal device (such as a smart toy or a smart learning machine) capable of executing the method provided in this application, which can effectively improve the user experience.

[0032] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0033] Reference Figure 1 , Figure 1 This is a flowchart illustrating a voice dialogue method provided in an embodiment of this application. The method includes, but is not limited to, steps S110 to S140: Step S110: Obtain the trained audio codec network and the user's first target speech signal; the trained audio codec network includes an audio encoding network and an audio decoding network; In this embodiment, the user's voice can be acquired using an audio acquisition device to obtain a first target speech signal. The audio codec network includes an audio encoding network and an audio decoding network, wherein the audio decoding network can specifically be a combination of several cascaded causal convolutional layers and Transformers.

[0034] Reference Figure 2 In some embodiments, the audio encoding network includes an encoding module and a quantization module, and the trained audio codec network is obtained through the following steps: Step S210: Obtain the first speech training group and the second speech training group; the first speech training group is a speech training group without audio labels; the second speech training group is a speech training group with audio labels. Step S220: Freeze the network parameters of the encoding module, and pre-train the quantization module and audio decoding network in the audio codec network according to the first speech training group to obtain the pre-trained quantization module and audio decoding network. Step S230: Unfreeze the network parameters of the encoding module, and fine-tune the encoding module, the trained quantization module, and the audio decoding network according to the second speech training group to obtain the trained audio codec network.

[0035] In this embodiment, the speech training group can be a combination of several speech training signals. The encoding module in the audio coding network can be a pre-trained Wav2Vec model as a self-supervised feature extractor, connected to a parallel tone prediction head, prosody prediction head, and Transformer bottleneck layer, respectively. The tone prediction head and prosody prediction head can be multilayer perceptrons (MLPs); and the Transformer bottleneck layer can include 8 Transformer layers. The quantization module can be a split residual vector quantization (Split-RVQ) structure.

[0036] Understandably, pre-training can involve freezing the model parameters of the pre-trained Wav2Vec model, then inputting the first speech training set into the audio codec network for self-supervised distillation pre-training, using the contrastive loss of the Wav2Vec model for initial semantic alignment, thereby obtaining the pre-trained quantization module and audio decoding network.

[0037] Fine-tuning training can involve unfreezing all parameters in the audio codec network and, based on the audio tags of the second speech training group, inputting the speech training signals from the second speech training group into the audio codec network for supervised fine-tuning training, thereby obtaining a trained audio codec network.

[0038] For example, the target loss function used for fine-tuning training can be expressed as:

[0039] in, The target loss function; For waveform reconstruction loss; For tone loss; For rhythmic loss; and This is an adjustable coefficient, and its specific value can be flexibly set according to the actual situation, such as 0.6, 0.4, etc.; The scale is the Fourier transform scale, such as 512, 1024, etc. The audio signal reconstructed from the output of the audio decoding network; This is the corresponding speech training signal in the second speech training group; This is a functional representation of the Sparse Fast Fourier Transform (SFFT). The predicted probability of the i-th tone output by the aforementioned tone prediction head, where the specific tone can be a neutral tone, a light tone, etc. This is an adjustable parameter, and its value can be 2; The prosodic vector of the speech training signal; The prosodic vector output by the aforementioned prosodic prediction head can specifically be a vector form of the signal accent, fundamental frequency, duration, and energy predicted by the prosodic prediction head.

[0040] Step S120: Input the first target speech signal into the audio coding network for signal encoding to obtain a first audio coding unit sequence. The first audio coding unit sequence includes a plurality of audio coding units, and each audio coding unit corresponds to an audio frame in the first target speech signal. In this embodiment of the application, the audio coding unit of the first target speech signal in each audio frame can be determined by an audio coding network to obtain the first audio coding unit sequence.

[0041] Reference Figure 3 In some embodiments, step S120, inputting the first target speech signal into the audio coding network for signal encoding to obtain a first audio coding unit sequence, includes: Step S310: Extract contextual features from the first target speech signal to obtain contextual embedding features; In this embodiment of the application, the context feature extraction can be performed by inputting the first target speech signal into the Wav2Vec model in the audio coding network, and processing the signal waveform of the first target speech signal into a context embedding through the Wav2Vec model, which is denoted as the context embedding feature.

[0042] Step S320: Perform multi-dimensional feature refinement on the context embedding features to obtain several audio coding units; Reference Figure 4 Furthermore, in step S320, the context embedding features are refined using multi-dimensional features to obtain several audio coding units, including: Step S410: Perform tone analysis on the context embedding features to obtain intermediate tone features; Step S420: Perform prosodic analysis on the context embedding features to obtain intermediate prosodic features; Step S430: Refine the semantic features of the context embedding features to obtain intermediate semantic features; Step S440: Obtain the audio encoding unit based on the intermediate tone feature, the intermediate prosody feature, and the intermediate semantic feature.

[0043] In this embodiment, multidimensional feature refinement may involve inputting the contextual embedding features output by the Wav2Vec model into a parallel tone prediction head, prosody prediction head, and Transformer bottleneck layer.

[0044] Specifically, for an audio coding unit corresponding to any audio frame of the first target speech signal, tone analysis can involve inputting context embedding features into a tone prediction head used to predict the tone of the speech signal to obtain intermediate tone features. These intermediate tone features can specifically be probability vectors representing any one of the following: a neutral tone, a soft tone, etc. Prosody analysis is similar to the tone analysis described above, obtaining a 4-dimensional vector output by the prosody prediction head. This 4-dimensional vector is in vector form representing the fundamental frequency, duration, energy, and stress of each audio frame of the target speech signal, and is denoted as intermediate prosodic features. Semantic feature refinement can be achieved by refining the semantic feature information of the context embedding features through a Transformer bottleneck layer combined with an EPAD / PAD mechanism to obtain intermediate semantic features. Step S440 can be a concatenation and integration of intermediate tone features, intermediate prosodic features, and intermediate semantic features to obtain the audio coding unit.

[0045] Step S330: Perform hierarchical residual vector quantization on all the audio coding units to obtain the first audio coding unit sequence.

[0046] In this embodiment, hierarchical residual vector quantization processing can input the audio coding units of each audio frame of the first target speech signal together into a quantization module with a hierarchical residual vector quantization structure to obtain several quantized audio coding units (tokens), and construct a first audio coding unit sequence based on all the quantized audio coding units. Step S130: Input the first target speech signal and the first audio coding unit sequence into the large language model for disambiguation inference to obtain the audio response unit sequence output by the large language model; In this embodiment, the large language model can be a large audio-language model (LALM), specifically a Qwen2-Audio large model; the first target speech signal and the first audio coding unit sequence can be input into the large language model together, and the large language model can perform disambiguation inference on the first target speech signal based on the first audio coding unit sequence to obtain the audio response unit sequence.

[0047] Reference Figure 5 In some embodiments, step S130, inputting the first target speech signal and the first audio coding unit sequence into a large language model for disambiguation inference to obtain the audio response unit sequence output by the large language model, includes: Step S510: Obtain the original text unit sequence corresponding to the first target speech signal; In this embodiment, the original text unit sequence may be a sequence set of text tokens of the first target language signal, which may be determined by a large language model based on the input first target speech signal.

[0048] Step S520: Based on the first audio coding unit sequence, perform semantic disambiguation on the original text unit sequence to obtain a text disambiguation unit sequence; Reference Figure 6 Further, step S520, performing semantic disambiguation on the original text unit sequence based on the first audio coding unit sequence to obtain a text disambiguation unit sequence, includes: Step S610: For the first audio coding unit sequence, the original text unit sequence is spliced ​​and projected to obtain multimodal embedding features; Step S620: Perform feature fusion and disambiguation processing on the multimodal embedding features to obtain the text disambiguation unit sequence.

[0049] In this embodiment, splicing projection can be achieved by splicing the first audio coding unit sequence and the original text unit sequence, and then aligning the spliced ​​unit sequence to the embedding space of the large language model through linear mapping. Specifically, the spliced ​​unit sequence is aligned to the embedding space of the Qwen2-Audio large model to obtain multimodal embedding features.

[0050] Feature fusion and disambiguation can be performed on the semantic backbone of the Qwen2-Audio large model to obtain multimodal fused features. This semantic backbone is the core Transformer of the Qwen2-Audio large model, which includes 32 Transformer layers. Then, the text tokens in the multimodal fused features are predicted by a multilayer perceptron (MLP) to assist in disambiguation, thereby obtaining a text disambiguation unit sequence.

[0051] Step S530: Perform an audio response on the text disambiguation unit sequence to obtain the audio response unit sequence.

[0052] In this embodiment, the audio response can be obtained by inputting a text disambiguation unit sequence into a causal temporal Transformer network, predicting the audio token sequence required for the audio response through the causal temporal Transformer network, thereby obtaining the audio response unit sequence. The parameters of this causal temporal Transformer network can be initialized based on the core Transformer of the Qwen2-Audio large model.

[0053] Step S140: Input the audio response unit sequence into the audio decoding network to reconstruct the signal waveform, obtain the voice response signal, and respond to the user's first target voice signal in dialogue according to the voice response signal.

[0054] In this embodiment, signal waveform reconstruction may be achieved by using an audio decoding network to reconstruct the audio token sequence in the audio response unit sequence into an audio signal, thereby obtaining a speech response signal, which is then input to a loudspeaker or other sound-emitting device for amplification to achieve a dialogue response.

[0055] Reference Figure 7 In some embodiments, the method further includes: Step S710: Obtain the audio mutation threshold and the second audio coding unit sequence output by the audio coding network in the dialogue response state; Step S720: Reconstruct the signal waveform of the second audio coding unit sequence to obtain the reconstructed signal; Step S730: Based on the audio mutation threshold, perform signal mutation detection on the reconstructed signal to obtain mutation detection results; Step S740: If the mutation detection result indicates the presence of a mutation, then a dialogue response is performed on the second target speech signal corresponding to the second audio coding unit sequence.

[0056] In this embodiment, the audio mutation threshold includes an energy mutation threshold and a fundamental frequency change threshold. The energy mutation threshold is used to characterize the maximum mutation threshold between two adjacent audio frames of the target speech signal, and its specific value can be any one of 10dBFS, 15dBFS, etc. The fundamental frequency change threshold is used to characterize the maximum change rate between two adjacent audio frames of the target speech signal, and its specific value can be any one of 50Hz / frame, 60Hz / frame, etc.

[0057] In practical applications, during the dialogue response (i.e., the speaker device broadcasts the sound according to the input voice response signal), the audio coding network can collect the second target voice signal provided by the user in real time and encode the signal to obtain the second audio coding unit sequence. The content of step S720 is similar to that of the aforementioned step S140 and can be easily deduced by analogy.

[0058] Understandably, signal mutation detection can be performed on the reconstructed signal based on an audio mutation threshold to obtain a mutation detection result. Specifically, if there is a frame energy mutation between two adjacent audio frames in the reconstructed signal that is greater than the energy mutation threshold, and the fundamental frequency change rate is greater than the fundamental frequency change threshold, then a mutation detection result indicating the presence of a mutation can be obtained. In this case, the dialogue response to the first target speech signal can be truncated, and a new speech response signal can be determined based on the second target speech signal and the second audio coding unit sequence to achieve a dialogue response to the second target speech signal. Conversely, if the mutation detection result indicates that no mutation exists, then the dialogue response to the first target speech signal can continue based on the speech response signal.

[0059] Figure 8 A system block diagram of a voice dialogue system provided in this application embodiment includes: The first processing unit 801 is used to acquire a trained audio codec network and the user's first target speech signal; the trained audio codec network includes an audio encoding network and an audio decoding network. The second processing unit 802 is used to input the first target speech signal into the audio coding network for signal encoding to obtain a first audio coding unit sequence. The first audio coding unit sequence includes a plurality of audio coding units, and each audio coding unit corresponds to an audio frame in the first target speech signal. The third processing unit 803 is used to input the first target speech signal and the first audio coding unit sequence into the large language model for disambiguation inference, and obtain the audio response unit sequence output by the large language model. The fourth processing unit 804 is used to input the audio response unit sequence into the audio decoding network for signal waveform reconstruction to obtain a voice response signal, and to perform a dialogue response to the user's first target voice signal based on the voice response signal.

[0060] It is worth mentioning that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0061] Figure 9 A schematic diagram of the structure of a computer device provided in this application embodiment includes: At least one processor 980; At least one memory 920 is used to store at least one program; When the at least one program is executed by the at least one processor 980, the at least one processor 980 performs the method as described in the foregoing embodiments.

[0062] This application also provides a computer-readable storage medium storing a processor-executable program, which, when executed by the processor 980, is used to implement the methods described in the foregoing embodiments.

[0063] Specifically, computer equipment can be either a user terminal or a server.

[0064] This application uses a computer device as a user terminal as an example, as detailed below: like Figure 9 As shown, the computer device 900 may include an RF (Radio Frequency) circuit 910, a memory 920 including one or more computer-readable storage media, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a WiFi module 970, a processor 980 including one or more processing cores, and a power supply 990, among other components. Those skilled in the art will understand that... Figure 9 The device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0065] The RF circuit 910 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 980 for processing; additionally, it transmits uplink data to the base station. Typically, the RF circuit 910 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. Furthermore, the RF circuit 910 can also communicate wirelessly with networks and other devices. Wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System for Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0066] The memory 920 can be used to store software programs and modules. The processor 980 executes various functional applications and data processing by running the software programs and modules stored in the memory 920. The memory 920 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device 900 (such as audio data, telephone directory, etc.). In addition, the memory 920 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 920 may also include a memory controller to provide access to the memory 920 by the processor 980 and the input unit 930. Although Figure 9 The RF circuit 910 is shown, but it is understood that it is not a necessary component of the computer device 900 and can be omitted as needed without changing the nature of the invention.

[0067] The input unit 930 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, the input unit 930 may include a touch-sensitive surface 932 and other input devices 931. The touch-sensitive surface 932, also known as a touch display screen or touchpad, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface 932), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 932 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 980, and can receive and execute commands from the processor 980. In addition, the touch-sensitive surface 932 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 932, the input unit 930 may also include other input devices 931. Specifically, other input devices 931 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0068] Display unit 940 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of computer device 900. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 940 may include display panel 941, optionally configured as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, touch-sensitive surface 932 may cover display panel 941. When touch-sensitive surface 932 detects a touch operation on or near it, it transmits the information to processor 980 to determine the type of touch event. Subsequently, processor 980 provides corresponding visual output on display panel 941 according to the type of touch event. Although in Figure 9 In this embodiment, the touch-sensitive surface 932 and the display panel 941 are implemented as two separate components to realize input and output functions. However, in some embodiments, the touch-sensitive surface 932 and the display panel 941 can be integrated to realize input and output functions.

[0069] The computer device 900 may also include at least one sensor 950, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 941 according to the ambient light level, and the proximity sensor can turn off the display panel 941 and / or backlight when the computer device 900 is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that the computer device 900 may also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0070] Audio circuitry 960, speaker 961, and microphone 962 provide an audio interface between the user and computer device 900. Audio circuitry 960 converts received audio data into electrical signals, which are then transmitted to speaker 961, where they are converted into sound signals for output. Conversely, microphone 962 converts collected sound signals into electrical signals, which are received by audio circuitry 960, converted back into audio data, and then processed by processor 980 before being transmitted via RF circuitry 910 to another control device, or output to memory 920 for further processing. Audio circuitry 960 may also include an earphone jack to facilitate communication between peripheral headphones and computer device 900.

[0071] Computer device 900 can transmit information with the wireless transmission module set up on the battle equipment via WiFi module 970.

[0072] The processor 980 is the control center of the computer device 900. It connects various parts of the control device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 920, and by calling data stored in the memory 920, it performs various functions of the computer device 900 and processes data, thereby providing overall monitoring of the control device. Optionally, the processor 980 may include one or more processing cores; optionally, the processor 980 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 980.

[0073] The computer device 900 also includes a power supply 990 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 980 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 990 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0074] Although not shown, the computer device 900 may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0075] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the methods described in the foregoing embodiments.

[0076] This application also discloses a computer program product or computer program, which includes computer instructions stored in the aforementioned computer-readable storage medium; the processor of the aforementioned electronic device can read the computer instructions from the aforementioned computer-readable storage medium, and the processor executes the computer instructions, causing the electronic device to perform the aforementioned method embodiment.

[0077] It is understood that the content of the above method embodiments is applicable to this computer program product or computer program embodiment. The specific functions implemented by this computer program product or computer program embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0078] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0079] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] The step numbers in the above method embodiments are set only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0085] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A voice dialogue method, characterized in that, include: The trained audio codec network and the user's first target speech signal are acquired; the trained audio codec network includes an audio encoding network and an audio decoding network; The first target speech signal is input into the audio coding network for signal encoding to obtain a first audio coding unit sequence. The first audio coding unit sequence includes a plurality of audio coding units, and each audio coding unit corresponds to an audio frame in the first target speech signal. The first target speech signal and the first audio coding unit sequence are input into a large language model for disambiguation inference to obtain the audio response unit sequence output by the large language model. The audio response unit sequence is input into the audio decoding network for signal waveform reconstruction to obtain a speech response signal, and a dialogue response is performed on the user's first target speech signal based on the speech response signal.

2. The method according to claim 1, characterized in that, The audio encoding network includes an encoding module and a quantization module. The trained audio encoding / decoding network is obtained through the following steps: Obtain a first speech training group and a second speech training group; the first speech training group is a speech training group without audio labels; the second speech training group is a speech training group with audio labels. Freeze the network parameters of the encoding module, and pre-train the quantization module and audio decoding network in the audio codec network according to the first speech training group to obtain the pre-trained quantization module and audio decoding network; Unfreeze the network parameters of the encoding module, and fine-tune the encoding module, the trained quantization module, and the audio decoding network according to the second speech training group to obtain the trained audio codec network.

3. The method according to claim 1, characterized in that, The step of inputting the first target speech signal into the audio coding network for signal encoding to obtain a first audio coding unit sequence includes: Contextual features are extracted from the first target speech signal to obtain contextual embedding features; The context embedding features are refined using multidimensional features to obtain several audio coding units; Hierarchical residual vector quantization is performed on all the audio coding units to obtain the first audio coding unit sequence.

4. The method according to claim 3, characterized in that, The multi-dimensional feature refinement of the context embedding features yields several audio coding units, including: Perform tone analysis on the context embedding features to obtain intermediate tone features; Prosodic analysis is performed on the context embedding features to obtain intermediate prosodic features; The context embedding features are semantically refined to obtain intermediate semantic features; The audio coding unit is obtained based on the intermediate tone feature, the intermediate prosodic feature, and the intermediate semantic feature.

5. The method according to claim 1, characterized in that, The step of inputting the first target speech signal and the first audio coding unit sequence into a large language model for disambiguation inference to obtain the audio response unit sequence output by the large language model includes: Obtain the original text unit sequence corresponding to the first target speech signal; Based on the first audio coding unit sequence, semantic disambiguation is performed on the original text unit sequence to obtain a text disambiguation unit sequence; The text disambiguation unit sequence is subjected to an audio response to obtain the audio response unit sequence.

6. The method according to claim 5, characterized in that, The step of performing semantic disambiguation on the original text unit sequence based on the first audio coding unit sequence to obtain a text disambiguation unit sequence includes: For the first audio coding unit sequence, the original text unit sequence is spliced ​​and projected to obtain multimodal embedding features; The multimodal embedding features are subjected to feature fusion and disambiguation processing to obtain the text disambiguation unit sequence.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Obtain the audio mutation threshold and the second audio coding unit sequence output by the audio coding network in the dialogue response state; The second audio coding unit sequence is reconstructed to obtain the reconstructed signal; Based on the audio mutation threshold, signal mutation detection is performed on the reconstructed signal to obtain mutation detection results; If the mutation detection result indicates the presence of a mutation, then a dialogue response is performed on the second target speech signal corresponding to the second audio coding unit sequence.

8. A voice dialogue system, characterized in that, include: The first processing unit is used to acquire a trained audio codec network and the user's first target speech signal; the trained audio codec network includes an audio encoding network and an audio decoding network. The second processing unit is used to input the first target speech signal into the audio coding network for signal encoding to obtain a first audio coding unit sequence. The first audio coding unit sequence includes a plurality of audio coding units, and each audio coding unit corresponds to an audio frame in the first target speech signal. The third processing unit is used to input the first target speech signal and the first audio coding unit sequence into the large language model for disambiguation inference, and obtain the audio response unit sequence output by the large language model. The fourth processing unit is used to input the audio response unit sequence into the audio decoding network for signal waveform reconstruction to obtain a voice response signal, and to perform a dialogue response to the user's first target voice signal based on the voice response signal.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images