Speech processing method, speech dialogue method and system, and computer terminal

Abstract

The present disclosure relates to the fields of artificial intelligence and speech interaction, and provides a speech processing method, a speech dialogue method and system, and a computer terminal. The speech processing method comprises: acquiring a first speech signal of a first interaction object at a current moment; encoding the first speech signal, and obtaining a first speech feature sequence; predicting a second speech feature sequence on the basis of the first speech feature sequence, a historical speech feature sequence, and a historical prediction feature sequence, the historical prediction feature sequence being used to represent a speech feature sequence predicted on the basis of the historical speech feature sequence; decoding the second speech feature sequence, and obtaining a second speech signal of a second interaction object at the current moment; and outputting the second speech signal to an interaction device of the first interaction object. The present disclosure solves the technical problems of relatively high latency and low accuracy in currently used speech interaction solutions.

Description

Speech processing methods, speech dialogue methods and systems, computer terminals Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and voice interaction, and more specifically, to a voice processing method, a voice dialogue method and system, and a computer terminal. Background Technology

[0002] Voice interaction technology can improve the convenience and efficiency of operation, especially in multitasking environments, freeing up users' hands, enabling personalized services and precise responses, and driving the development of fields such as virtual companionship, smart homes, customer service systems, and medical assistants. Full-duplex voice dialogue is a typical application of voice interaction technology. In full-duplex voice interaction communication, both parties can communicate simultaneously without interfering with each other. This differs from half-duplex communication, which allows one party to speak at a time, with the other responding afterwards. In full-duplex voice dialogue, both parties can communicate in real time, improving the fluency and naturalness of communication.

[0003] Currently, full-duplex voice interaction solutions mainly achieve interaction between users and voice tools through cascaded systems. However, since data is forwarded between multiple nodes, delays may be introduced, which may affect real-time performance. Furthermore, since the algorithms in multiple stages may have errors, cascaded systems generally amplify the errors in a single stage, which can significantly reduce the overall dialogue quality and result in low accuracy of voice interaction.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This disclosure provides a voice processing method, a voice dialogue method and system, and a computer terminal to at least solve the technical problems of high latency and low accuracy in commonly used voice interaction solutions.

[0006] According to one aspect of the present disclosure, a voice processing method is provided, comprising: acquiring a first voice signal of a first interactive object at a current time; encoding the first voice signal to obtain a first voice feature sequence; predicting a second voice feature sequence based on the first voice feature sequence, a historical voice feature sequence, and a historical predicted feature sequence, wherein the historical voice feature sequence is used to characterize the voice feature sequence corresponding to the voice signal sent by the first interactive object at a historical time, and the historical predicted feature sequence is used to characterize the voice feature sequence predicted based on the historical voice feature sequence; decoding the second voice feature sequence to obtain a second voice signal of the second interactive object at the current time; and outputting the second voice signal to an interactive device of the first interactive object.

[0007] According to another aspect of the embodiments of this disclosure, a voice dialogue method is also provided, comprising: acquiring a user voice signal at the current moment; encoding the user voice signal to obtain a first voice feature sequence of the user voice signal; predicting a second voice feature sequence based on the first voice feature sequence, a historical voice feature sequence, and a historical predicted feature sequence, wherein the historical voice feature sequence is used to characterize the voice feature sequence corresponding to the user voice signal acquired at a historical moment, and the historical predicted feature sequence is used to characterize the voice feature sequence predicted based on the historical voice feature sequence; decoding the second voice feature sequence to obtain a response voice signal at the current moment; and playing the response voice signal.

[0008] According to another aspect of the embodiments of this disclosure, a voice processing method is also provided, comprising: obtaining a first voice signal of a first interactive object at the current moment by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the first voice signal, and the first voice signal is used to characterize the voice signal of the first interactive object at the current moment during the voice interaction process; encoding the first voice signal to obtain a first voice feature sequence of the first voice signal; predicting a second voice feature sequence based on the first voice feature sequence, a historical voice feature sequence, and a historical predicted feature sequence, wherein the historical voice feature sequence is used to characterize the voice feature sequence corresponding to the voice signal sent by the first interactive object at a historical moment, and the historical predicted feature sequence is used to characterize the voice feature sequence predicted based on the historical voice feature sequence; decoding the second voice feature sequence to obtain a second voice signal of the second interactive object at the current moment, wherein the second voice signal is used to characterize the voice signal of the second interactive object at the current moment during the voice interaction process; and outputting the second voice signal by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter includes the second voice signal.

[0009] According to another aspect of the embodiments of this disclosure, a voice dialogue system is also provided, including: a sound acquisition and encoding module, configured to acquire a user's voice signal at the current moment and encode the user's voice signal to obtain a first voice feature sequence; a multimodal processing module, configured to predict a second voice feature sequence based on the first voice feature sequence, a historical voice feature sequence, and a historical predicted feature sequence, wherein the historical voice feature sequence is used to characterize the voice feature sequence corresponding to the voice signal sent by the first interactive object at a historical moment, and the historical predicted feature sequence is used to characterize the voice feature sequence predicted based on the historical voice feature sequence; and a voice decoding and playback module, configured to decode the second voice feature sequence to obtain a response voice signal at the current moment and play the response voice signal.

[0010] According to another aspect of the embodiments of this disclosure, a voice dialogue system is also provided, including: a acquisition device configured to acquire a user's voice signal at the current moment; a voice encoding module configured to encode the user's voice signal to obtain a first voice feature sequence; a multimodal processing module configured to predict a second voice feature sequence based on the first voice feature sequence, a historical voice feature sequence, and a historical predicted feature sequence, wherein the historical voice feature sequence is used to characterize the voice feature sequence corresponding to a voice signal sent by a first interactive object at a historical moment, and the historical predicted feature sequence is used to characterize the voice feature sequence predicted based on the historical voice feature sequence; a voice decoding module configured to decode the second voice feature sequence to obtain a response voice signal at the current moment; and a playback device configured to play the response voice signal.

[0011] According to another aspect of the embodiments of this disclosure, a computer terminal is also provided, including: a memory storing an executable program; and a processor configured to run the program, wherein the program executes the methods in the various embodiments of this disclosure when it runs.

[0012] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of the various embodiments of the present disclosure.

[0013] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.

[0014] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.

[0015] According to another aspect of the embodiments of this disclosure, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this disclosure.

[0016] In this embodiment, a first voice signal of the first interactive object at the current moment is acquired; the first voice signal is encoded to obtain a first voice feature sequence; a second voice feature sequence is predicted based on the first voice feature sequence, historical voice feature sequences, and historical predicted feature sequences; the second voice feature sequence is decoded to obtain a second voice signal of the second interactive object at the current moment; and the second voice signal is output to the interactive device of the first interactive object. By encoding and decoding the voice signal, the conversion between the voice signal and the voice feature sequence is realized, enabling a single node to predict the voice signal based on multiple feature sequences such as the first voice feature sequence, historical voice feature sequences, and historical predicted feature sequences. This avoids the delay introduced by forwarding the voice feature sequence between multiple nodes, and thus avoids the error superposition caused by data processing by multiple nodes. This achieves the technical effect of reducing voice interaction latency and improving voice interaction accuracy, thereby solving the technical problems of high latency and low accuracy in commonly used voice interaction schemes.

[0017] The above general description and the following detailed description are for illustrative and explanatory purposes of this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain this disclosure. In the drawings:

[0019] Figure 1 is a schematic diagram of an application scenario of a speech processing method according to an embodiment of the present disclosure;

[0020] Figure 2 is a structural block diagram of a computing environment for a speech processing method according to an embodiment of the present disclosure;

[0021] Figure 3 is a flowchart of a speech processing method according to an embodiment of the present disclosure;

[0022] Figure 4 is a schematic diagram of an optional encoding process for a first speech signal according to an embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram of an optional decoding process for a second speech feature sequence according to an embodiment of the present disclosure;

[0024] Figure 6 is a schematic diagram of predicting a second speech feature sequence when an optional historical prediction feature sequence includes a predicted speech feature sequence, according to an embodiment of the present disclosure;

[0025] Figure 7 is a schematic diagram of predicting a second speech feature sequence and a text feature sequence when an optional historical prediction feature sequence includes a predicted speech feature sequence and a predicted text feature sequence according to an embodiment of the present disclosure;

[0026] Figure 8 illustrates an optional full-duplex voice communication method based on a voice processing method according to an embodiment of the present disclosure.

[0027] Figure 9 is a flowchart of a voice dialogue method according to an embodiment of the present disclosure;

[0028] Figure 10 is a flowchart of a speech processing method according to an embodiment of the present disclosure;

[0029] Figure 11 is a schematic diagram of a voice dialogue system according to an embodiment of the present disclosure;

[0030] Figure 12 is a schematic diagram of a voice dialogue system according to an embodiment of the present disclosure;

[0031] Figure 13 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0033] The terms "first," "second," etc., used in this disclosure, claims, and accompanying drawings are for distinguishing similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:

[0035] Full-duplex voice communication refers to a scenario where both parties can communicate simultaneously without interfering with each other. This differs from half-duplex communication, which allows one party to speak at a given time while the other responds later. In full-duplex voice conversations, both parties can communicate in real time, enhancing the fluency and naturalness of the communication.

[0036] Multi-channel modeling: refers to a modeling method that processes the input and output of multiple channels simultaneously to achieve bidirectional communication.

[0037] Modal alignment: refers to learning to align data of different modalities (such as speech, text, etc.) during model processing in order to improve the model's ability to handle multimodal scenarios.

[0038] Input voice channel: refers to the input signal provided by the user's voice to the system in a voice dialogue scenario.

[0039] Output voice or text channel: refers to the output signal provided by the system to the user through voice or text in a voice dialogue scenario.

[0040] Timing alignment: refers to the ability of the model to effectively align the timing signals of the input and output speech during processing, thereby achieving a full-duplex speech dialogue mode.

[0041] According to an embodiment of this disclosure, an application scenario for a voice processing method is provided. The steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0042] Considering the significant computational resource consumption of machine learning models on mobile terminals, Figure 1 is a schematic diagram of an application scenario of a speech processing method according to an embodiment of this disclosure. The method provided by this embodiment can be applied to the application scenario shown in Figure 1. In the application scenario shown in Figure 1, the machine learning model is deployed on server 10. Server 10 can connect to one or more clients 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Clients 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Clients 20 can interact with users through a graphical user interface to invoke the large model, thereby implementing the method provided by this embodiment.

[0043] If the client device's operating resources are sufficient to meet the deployment and operation requirements of a large model, the embodiments disclosed herein can be performed on the client device.

[0044] Figure 2 is a structural block diagram of a computing environment for a speech processing method according to an embodiment of the present disclosure. As shown in Figure 2, the computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ... in the figure). Each computing node contains local processing and memory resources, and the end user 202 can remotely run applications or store data within the computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within the computing environment 201, representing services "A", "D", "E", and "H", respectively.

[0045] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 201).

[0046] The services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. VM-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0047] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in Figure 2, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod handle requests related to one or more corresponding functions of the service. The proxy 245 typically controls service-related network functions such as routing and load balancing. Other services can also be equipped with similar Pods.

[0048] During operation, executing a user request from end user 202 may require calling one or more services in computing environment 201. Executing one or more functions of one service requires calling one or more functions of another service. As shown in Figure 2, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to execute one or more functions.

[0049] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0050] In the aforementioned operating environment, this disclosure provides a voice processing method as shown in Figure 3. This voice processing method can be executed by the server in the embodiment shown in Figure 1. One optional application of the voice processing method shown in Figure 3 is an intelligent legal assistant. When a user asks a question to the intelligent legal assistant, the assistant can use the voice processing method to provide legal advice, thereby reducing legal risks and improving user work efficiency. The intelligent legal assistant applying the voice processing method can also be used for internal employee training and legal knowledge dissemination, improving employees' legal literacy and risk awareness.

[0051] Figure 3 is a flowchart of a speech processing method according to an embodiment of the present disclosure. As shown in Figure 3, the method may include the following steps:

[0052] Step S302: Obtain the first voice signal of the first interactive object at the current moment.

[0053] The first interaction object in the above steps is the party that sends voice signals during the voice interaction process in a full-duplex voice dialogue scenario. This party can be a human user interacting with a voice assistant or other voice interaction devices.

[0054] The first speech signal in the above steps is the speech signal emitted by the first interactive object at the current moment during the speech interaction process. The speech signal is formed by sound waves and can contain information such as speech content, tone, and speech rate.

[0055] In one optional embodiment, when the voice processing method is executed by a mobile terminal such as a mobile phone or laptop, the voice input of the first interactive object can be directly acquired through a microphone as the first voice signal. When the voice processing method is executed by a server, the voice signal of the first interactive object can be acquired through a client, and the client can send the voice signal to the server, using the voice signal sent by the client as the first voice signal. The first voice signal may or may not contain voice content, and both the first voice signal and the voice signal used to respond to the first voice signal can be a silence signal.

[0056] Step S304: Encode the first speech signal to obtain the first speech feature sequence.

[0057] The first speech feature sequence in the above steps is a series of feature sequences obtained by encoding the continuous first speech signal.

[0058] In one alternative embodiment, a pre-trained encoding model can be used to process the first speech signal. The first speech signal is input into the trained encoding model, which encodes the first speech signal and outputs a first speech feature sequence. The encoding model used for encoding processing needs to perform fast encoding processing on the first speech feature sequence to support the real-time requirements of full-duplex voice communication.

[0059] The first voice signal is processed by encoding, avoiding a cumbersome judgment process. Both the first voice signal and the voice signal used to respond to the first voice signal can be silence signals, which enables real-time processing of the first voice signal and ensures the real-time performance of full-duplex voice communication.

[0060] For example, FIG4 is a schematic diagram of an optional encoding process for a first speech signal according to an embodiment of the present disclosure. As shown in FIG4, the user has a conversation with a voice assistant through the microphone on the headset. Based on the first speech signal collected by the microphone of the headset, the first speech signal is encoded to obtain a first speech feature sequence 40, which includes a plurality of features 42.

[0061] Step S306: Based on the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence, a second speech feature sequence is predicted. The historical speech feature sequence is used to characterize the speech feature sequence corresponding to the speech signal sent by the first interactive object at a historical time, and the historical predicted feature sequence is used to characterize the speech feature sequence predicted based on the historical speech feature sequence.

[0062] The historical speech feature sequence in the above steps refers to the features recorded at past time points, used to characterize the speech signal sent by the first interactive object at a past moment.

[0063] The historical prediction feature sequence in the above steps is a feature sequence predicted based on the historical speech feature sequence. The speech signal generated based on the historical prediction feature sequence can be used to respond to the speech signal sent by the first interactive object at a past moment.

[0064] Both historical speech feature sequences and historical predicted feature sequences are historical data. Historical speech feature sequences characterize the speech signals emitted by human users in the past, while historical predicted feature sequences characterize the speech signals responded to by voice assistants or other voice interaction devices by human users in the past. These historical and predicted feature sequences can provide a reference when predicting the second speech feature sequence, thereby improving the accuracy of the prediction.

[0065] The second speech feature sequence in the above steps can characterize the features of the response speech signal used to respond to the first speech signal, and can be used to generate a speech signal that responds to the first speech signal.

[0066] In one optional embodiment, when predicting the speech signal of the second interactive object, a neural network model can be used to complete the prediction task. The neural network model can be a Long Short-Term Memory (LSTM) model, a Gated Recurrent Unit (GRU) model, or a Large Language Model (LLM). The first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence are input into the trained neural network as input sequences. By learning the correlations between the input sequences and combining them with the context, the neural network can predict the second speech feature sequence.

[0067] In another optional embodiment, the mapping rule between the speech feature sequence and the predicted feature sequence can be determined based on the correspondence between the historical speech feature sequence and the historical predicted feature sequence in terms of features such as spectral features, energy features, and speech rate features. Then, based on the first speech feature sequence, the mapping rule between the speech feature sequence and the predicted feature sequence is used to predict the second speech feature sequence.

[0068] Step S308: Decode the second speech feature sequence to obtain the second speech signal of the second interactive object at the current time.

[0069] The second interactive object in the above steps is the party that receives and responds to voice signals during the voice interaction process. It can be a voice assistant or other voice interaction device that interacts with human users.

[0070] The second speech signal in the above steps corresponds to the first speech signal and is a speech response to the first speech signal generated based on the first speech signal.

[0071] In one alternative embodiment, a predefined audio processing algorithm can be used to decode the second speech feature sequence to obtain sound signal waveform data, i.e., the second speech signal. The audio processing algorithm used for decoding needs to perform fast decoding of the second speech feature sequence to support the real-time requirements of full-duplex voice communication.

[0072] By processing the second speech feature sequence through decoding, real-time processing of the second speech feature sequence can be achieved, ensuring the real-time performance of full-duplex voice communication.

[0073] Figure 5 is a schematic diagram of an optional decoding process of a second speech feature sequence according to an embodiment of the present disclosure. As shown in Figure 5, the second speech feature sequence 50 containing multiple features is decoded to obtain a second speech signal in the form of a speech signal. Then, the second speech signal is played using a speaker to achieve the purpose of responding to the first speech signal.

[0074] Step S310: Output the second voice signal to the interactive device of the first interactive object.

[0075] The interactive device mentioned above is used to interact with the first interactive object. The interactive device can play a first voice signal in response to the first voice signal emitted by the first interactive object. The interactive device can be a mobile phone, a laptop computer, a voice assistant, or other voice interaction devices.

[0076] In one optional embodiment, when the interaction device is a mobile terminal such as a mobile phone or laptop, and the entity executing the voice processing method is also a mobile terminal, the second voice signal can be sent to the speaker of the mobile terminal and played through the speaker. When the interaction device is a client such as a voice assistant, and the entity executing the voice processing method is a server, the second voice signal can be transmitted from the server back to the client, whereby the client processes and plays the second voice signal. Processing the second voice signal allows the first interactive object that sent the first voice signal to receive a response to the first voice signal, thus achieving the purpose of voice interaction.

[0077] In this embodiment, a first voice signal of the first interactive object at the current moment is acquired; the first voice signal is encoded to obtain a first voice feature sequence; a second voice feature sequence is predicted based on the first voice feature sequence, historical voice feature sequences, and historical predicted feature sequences; the second voice feature sequence is decoded to obtain a second voice signal of the second interactive object at the current moment; and the second voice signal is output to the interactive device of the first interactive object. By encoding and decoding the voice signal, the conversion between the voice signal and the voice feature sequence is realized, enabling a single node to predict the voice signal based on multiple feature sequences such as the first voice feature sequence, historical voice feature sequences, and historical predicted feature sequences. This avoids the delay introduced by forwarding the voice feature sequence between multiple nodes, and thus avoids the error superposition caused by data processing by multiple nodes. This achieves the technical effect of reducing voice interaction latency and improving voice interaction accuracy, thereby solving the technical problems of high latency and low accuracy in commonly used voice interaction schemes.

[0078] In the above embodiments of this disclosure, a second speech feature sequence is predicted based on a first speech feature sequence, a historical speech feature sequence, and a historical predicted feature sequence, including: inputting the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence into a multimodal processing model, and using the multimodal processing model to predict the second speech feature sequence.

[0079] The multimodal processing model in the above steps is a model used to predict the second speech feature sequence, which can be a text-speech pretrained transformer model. The text-speech pretrained transformer model learns the mapping relationship between text and speech by pretraining on large-scale text and speech data, and can complete the task of converting text to speech.

[0080] In one optional embodiment, the first speech feature sequence and the historical speech feature sequence can be in text form. The historical speech feature sequence and the historical predicted feature sequence are input into the multimodal processing model to provide historical data reference for the prediction of the second speech feature sequence, thereby increasing the prediction accuracy of the second speech feature sequence. Simultaneously, the text-based first speech feature sequence is also input into the multimodal processing model, enabling the model to understand the intent of the first interactive object contained in the first speech feature sequence. Combining this with the context and referring to the correspondence between the historical speech feature sequence and the historical predicted feature sequence, the model generates a suitable response text, i.e., the second speech feature sequence.

[0081] In the above embodiments of this disclosure, the historical predicted feature sequence includes at least one type of feature sequence; inputting the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence into a multimodal processing model includes: inputting the first speech feature sequence and the historical speech feature sequence into a first channel of the multimodal processing model, wherein the first channel is used to input external speech feature sequences into the multimodal processing model; inputting feature sequences of at least one type into at least one second channel of the multimodal processing model, wherein the at least one second channel corresponds one-to-one with at least one type, wherein the second channel is used to input feature sequences generated by the multimodal processing model into the multimodal processing model.

[0082] The first channel in the above steps is the input channel of the multimodal processing model, used to input the first speech feature sequence and the historical speech feature sequence. The first speech feature sequence and the historical speech feature sequence are external speech feature sequences.

[0083] The second channel in the above steps is also an input channel of the multimodal processing model, used to input the historical predicted feature sequence, which is a feature sequence generated by the multimodal processing model. The number of second channels is equal to the number of feature sequence types contained in the historical predicted feature sequence; one second channel is used to input a feature sequence of one type from the historical predicted feature sequence.

[0084] In one optional embodiment, the historical prediction feature sequence may contain one or more feature sequences of different types. However, to achieve speech signal prediction, the historical prediction feature sequence should at least contain a speech-type feature sequence. For example, the feature sequence may contain only one speech type, or it may include both speech and text types. The types of feature sequences contained in the historical prediction feature sequence determine the number of second channels.

[0085] In the process of inputting the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence into the multimodal processing model, different channels are used for input. The first channel is used to input the first speech feature sequence and the historical speech feature sequence, and the second channel is used to input the historical predicted feature sequence. Each second channel can input one type of feature data from the historical predicted feature sequence. For example, if the historical predicted feature sequence contains one type of feature sequence, there is one second channel, and the modeling scheme of the multimodal processing model can be considered a two-channel modeling scheme, including one first channel and one second channel; if the historical predicted feature sequence contains two types of feature sequences, there are two second channels, and the modeling scheme of the multimodal processing model can be considered a three-channel modeling scheme, including one first channel and two second channels.

[0086] In the above embodiments of this disclosure, at least one type is determined based on one of the following information: network environment data of the voice interaction process, output requirements carried in the first voice signal, and output requirements received before the start of the voice interaction process, wherein the output requirements are used to characterize the type of data that the second interaction object needs to output at the current moment.

[0087] The network environment data in the above steps may include, but is not limited to, network latency, packet loss rate, etc., and the network environment data can be used to determine the type of feature sequence in the historical prediction feature sequence.

[0088] The output requirements in the above steps are the requirements issued by the user during the voice interaction dialogue or before the voice interaction begins. They represent the data type of the second interactive object specified by the user at the current moment. The data type of the second interactive object's output at the current moment can be either voice or text.

[0089] In an optional embodiment, the type of feature sequence in the historical prediction feature sequence can be determined by network environment data. For example, if the historical prediction feature sequence contains multiple types of feature sequences when the network latency is high and the packet loss rate is high, it will put greater transmission pressure on the network. Therefore, the type of feature sequence in the historical prediction feature sequence is based on the speech type.

[0090] In another optional embodiment, the type of the feature sequence in the historical predicted feature sequence can determine the type of the feature sequence in the second speech feature sequence. For example, if the type of the feature sequence in the historical predicted feature sequence includes text and speech, then the type of the feature sequence in the second speech feature sequence also includes text and speech. A voice assistant or other voice interaction device can output text or play a second speech signal based on the second speech feature sequence. If the type of the feature sequence in the historical predicted feature sequence includes speech, then the type of the feature sequence in the second speech feature sequence is also speech. A voice assistant or other voice interaction device can play a second speech signal based on the second speech feature sequence.

[0091] The type of feature sequence in the historical predicted feature sequence can also be determined by the output request carried in the first speech signal. During the voice interaction dialogue, the user can issue an output request, specifying whether the voice assistant should respond to the first speech signal issued by the user in text or speech form. For example, if the user specifies that the voice assistant should respond in both text and speech form during the voice interaction dialogue, the type of feature sequence in the historical predicted feature sequence can be determined to be text type and speech type.

[0092] In another optional embodiment, the type of feature sequence in the historical predicted feature sequence can also be determined by the output request received before the start of the voice interaction process. Before the start of the voice interaction process, the user can issue an output request via an interaction button with the voice assistant, specifying whether the voice assistant should respond to the user's first voice signal in text or voice format. For example, if the output request indicates that the user specifies the voice assistant to respond in voice format, then the type of feature sequence in the historical predicted feature sequence can be determined to be voice type.

[0093] In the above embodiments of this disclosure, the historical prediction feature sequence includes at least a predicted speech feature sequence, wherein different features in the predicted speech feature sequence are used to characterize different signal elements in the speech signal of the second interactive object at a historical moment.

[0094] The predicted speech feature sequence in the above steps is a sequence of speech features from the historical predicted feature sequences.

[0095] In one optional embodiment, the historical predicted feature sequence includes at least the predicted speech feature sequence. Therefore, the second speech feature sequence predicted based on the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence includes at least the speech type feature sequence. Processing the second speech feature sequence yields at least the second speech signal.

[0096] Figure 6 is a schematic diagram of predicting a second speech feature sequence when the optional historical predicted feature sequence includes a predicted speech feature sequence, according to an embodiment of the present disclosure. As shown in Figure 6, the first speech feature sequence 60 and the historical speech feature sequence 62 are input to the multimodal processing model through the first channel, and the predicted speech feature sequence 64 is input to the multimodal processing model through the second channel. The multimodal processing model predicts the second speech feature sequence 66 based on the first speech feature sequence 60, the historical speech feature sequence 62, and the predicted speech feature sequence 64. Each of the first speech feature sequence 60, the historical speech feature sequence 62, the predicted speech feature sequence 64, and the second speech feature sequence 66 contains multiple features. Figure 6 uses an example where each of these sequences contains 5 features, but the actual number of features in practical applications may not be limited to this.

[0097] In the above embodiments of this disclosure, the historical prediction feature sequence further includes a predicted text feature sequence, wherein different features in the predicted text feature sequence are used to characterize different text elements of the historical predicted text, and the historical predicted text is used to characterize the text corresponding to the speech signal of the second interactive object at a historical moment; the method further includes: decoding the text feature sequence output by the multimodal processing model to obtain the target text, wherein the target text is used to characterize the text corresponding to the speech signal of the second interactive object at the current moment; and outputting the target text to the interactive device of the first interactive object.

[0098] The predicted text feature sequence in the above steps is a sequence of text-type feature sequences from the historical predicted feature sequences.

[0099] The text feature sequence in the above steps is a feature of the text used to respond to the first speech signal.

[0100] The target text in the above steps is the text response to the first speech signal that corresponds to the first speech signal.

[0101] In one optional embodiment, during voice interaction, the response generated by the second interaction object can be in voice form or text form. That is, at a historical moment, the response generated by the second interaction object includes both the predicted voice signal of the historical moment and the historical predicted text. Different text elements in the historical predicted text are represented by the predicted text feature sequence in the historical predicted feature sequence.

[0102] When the historical predicted feature sequence includes both predicted speech feature sequence and predicted text feature sequence, a multimodal processing model is used to predict the text feature sequence based on the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence. Then, a text generation model can be used to convert the text feature sequence into target text. The text generation model can be a recurrent neural network, a long short-term memory network, etc. The generated target text can be displayed to the first interactive object on the screen of the interactive device, thus achieving the purpose of responding to the first speech signal emitted by the first interactive object.

[0103] For example, Figure 7 is a schematic diagram of predicting a second speech feature sequence and a text feature sequence when an optional historical prediction feature sequence includes a predicted speech feature sequence and a predicted text feature sequence according to an embodiment of the present disclosure. As shown in Figure 7, the predicted speech feature sequence 70 in the historical prediction feature sequence is input into the multimodal processing model through a second channel, the predicted text feature sequence 72 in the historical prediction feature sequence is input into the multimodal processing model through another second channel, and the first speech feature sequence 74 and the historical speech feature sequence 76 are input into the multimodal processing model through a first channel. By performing prediction through the multimodal processing model, a text feature sequence 79 can be obtained simultaneously with the second speech feature sequence 78. Discretization decoding of the second speech feature sequence 78 can obtain a second speech signal, and decoding of the text feature sequence 79 can obtain the target text. The target text and the second speech signal have the same content, but different presentation formats; the target text is presented in text form, and the second speech signal is presented in speech form. Among them, the predicted speech feature sequence 70, the predicted text feature sequence 72, the first speech feature sequence 74, the historical speech feature sequence 76, the second speech feature sequence 78, and the text feature sequence 79 each contain multiple features. Figure 6 takes the example that the predicted speech feature sequence 70, the first speech feature sequence 74, the historical speech feature sequence 76, and the second speech feature sequence 78 each contain 5 features, and the predicted text feature sequence 72 and the text feature sequence 79 each contain 2 features. However, the number of features included in actual applications may not be limited to these.

[0104] In the above embodiments of this disclosure, encoding a first speech signal to obtain a first speech feature sequence includes: performing discretization encoding processing on the first speech signal to obtain a first speech feature sequence, wherein different features in the first speech feature sequence are used to characterize different signal elements in the first speech signal; decoding a second speech feature sequence to obtain a second speech signal of the second interactive object at the current time includes: performing discretization decoding processing on the second speech feature sequence to obtain a second speech signal, wherein different features in the first speech feature sequence are used to characterize different signal elements in the first speech signal.

[0105] The signal elements in the above steps are the basic units in the first speech signal, such as phonemes, syllables, and speech rate. That is, the first speech feature sequence describes the characteristics of the basic units such as phonemes, syllables, and speech rate in the first speech signal in the form of discrete feature sequences.

[0106] In one optional embodiment, when discretizing and encoding the first speech signal, the first speech signal is divided into frames, and then each frame obtained after framing is windowed to reduce signal jumps at frame boundaries. Feature extraction is performed on the windowed frames. Features may include, but are not limited to, short-time energy, zero-crossing rate, and Mel-Frequency Cepstral Coefficients (MFCCs). The short-time energy characterizes the energy level of the first speech signal, the zero-crossing rate characterizes the frequency characteristics of the first speech signal, and the MFCCs characterize the spectral characteristics of the first speech signal. The extracted features are then encoded, converting continuous feature values ​​into discrete encoded sequences to obtain the first speech feature sequence. The encoding method can be quantization, which maps continuous feature values ​​to a finite set of discrete values; or it can use Discrete Cosine Transform (DCT) to encode the feature signal, taking the first few coefficients as feature representations; or it can use Licensed Professional Counselor (LPC) encoding, obtaining linear prediction coefficients through linear prediction analysis to represent the features of the speech signal.

[0107] Discrete feature values ​​from the second speech feature sequence can be input into the decoding model. The decoding model can then perform discretization decoding on the second speech feature sequence to obtain the second speech signal.

[0108] In the above embodiments of this disclosure, the first speech signal is discretized and encoded to obtain a first speech feature sequence, including: using an encoder to extract features from the first speech signal to obtain a first speech vector corresponding to the first speech signal; using a quantizer to quantize the first speech vector through multiple preset features to obtain a first speech feature sequence; and the second speech feature sequence is discretized and decoded to obtain a second speech signal, including: using a decoder to decode the second speech feature sequence to obtain a second speech signal.

[0109] The encoder mentioned in the above steps is a device or algorithm used to convert an input signal into a specific form. In speech processing, the encoder is used to extract features from a speech signal, converting the first speech signal into the form of a first speech vector.

[0110] The first speech vector in the above steps is a vector representing the speech signal obtained by feature extraction from the first speech signal by the encoder. It can be used to generate the first speech feature sequence.

[0111] The quantizer mentioned in the above steps is a device or algorithm that converts a continuous signal into a discrete signal. In speech processing, a quantizer can convert a continuous first speech vector into a discrete first speech feature sequence.

[0112] A decoder is a device or algorithm that converts data in a specific form back into raw data. In speech processing, a decoder decodes a second speech feature sequence, converting the discrete speech feature sequence into a second speech signal.

[0113] In one optional embodiment, a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN) can be selected as the encoder, but the choice of encoder is not limited to these. The first speech signal is input into the encoder, which extracts features from the first speech signal and outputs a first speech vector corresponding to the first speech signal. The obtained first speech feature vector is input into a quantizer, which can be an algorithm that maps continuous values ​​to discrete values. The quantizer quantizes the feature vector using multiple preset features, such as speech content, pitch, and speech rate. After quantizing the feature vector, the quantizer outputs a first speech feature sequence.

[0114] The discrete second speech feature sequence is input into the decoder, which decodes the input second speech feature sequence to generate the second speech signal. The decoder can be a statistical model-based decoder, such as a Hidden Markov Model (HMM) or Dynamic Time Warping (DTW), or a neural network-based decoder, such as an RNN or a Long Short-Term Memory Network (LSTM).

[0115] The following description uses a preferred embodiment. Figure 8 illustrates an optional full-duplex voice communication method based on a speech processing approach according to an embodiment of this disclosure. As shown in Figure 8, during full-duplex voice communication, sound acquisition and discretization encoding are performed first. Specifically, the system can acquire the user's voice input via a microphone as a first voice signal. The first voice signal is then discretized and encoded. The first voice feature sequence obtained from the discretization encoding is then input into a multimodal processing model. Specifically, a Large Language Model (LLM) can be selected as the multimodal processing model to understand the user's intent and generate a second voice feature sequence. The second voice feature sequence is then discretized and decoded to obtain a second voice signal, which is prepared for output through a speaker. The second voice signal is played through the speaker to provide feedback to the user. By using discretization encoding and discretization decoding, i.e., voice discretization decoding and playback, the cumbersome judgment process for the first voice signal is avoided, and the transmission of the voice signal between different modules is reduced. This enables real-time processing of the first voice signal and real-time feedback of the second voice signal, ensuring the real-time performance of full-duplex voice communication. Full-duplex voice communication methods can also use a detection module to monitor the user's state during a full-duplex conversation. Once the user hears the system's voice feedback, they can interrupt and start speaking at any time. The detection module is always active, ensuring it can capture the user's voice input at any time, determine if the user is interrupting, and enable the system to quickly switch between each user's speech and response, achieving a natural and fluent dialogue.

[0116] According to embodiments of this disclosure, a voice dialogue method is also provided, wherein the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be performed in a different order than that shown here.

[0117] Figure 9 is a flowchart of a voice dialogue method according to an embodiment of the present disclosure. As shown in Figure 9, the method includes the following steps:

[0118] Step S902: Collect the user's voice signal at the current moment;

[0119] Step S904: Encode the user's speech signal to obtain the first speech feature sequence of the user's speech signal;

[0120] Step S906: Based on the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence, a second speech feature sequence is predicted. The historical speech feature sequence is used to characterize the speech feature sequence corresponding to the user speech signal collected at a historical time, and the historical predicted feature sequence is used to characterize the speech feature sequence predicted based on the historical speech feature sequence.

[0121] Step S908: Decode the second speech feature sequence to obtain the response speech signal at the current moment;

[0122] Step S910: Play the response voice signal.

[0123] The mobile terminal is the executing entity of the above-described voice dialogue method. The user voice signal in the above steps refers to the voice signal emitted by the user, which may include voice commands, dialogue content, etc. The response voice signal refers to the corresponding response made by the system or device based on the user's voice signal, which may include voice prompts, answers, reminders, etc.

[0124] According to embodiments of this disclosure, a voice processing method is also provided, wherein the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0125] Figure 10 is a flowchart of a speech processing method according to an embodiment of the present disclosure. As shown in Figure 10, the method includes the following steps:

[0126] Step S1002: Obtain the first voice signal of the first interactive object at the current moment by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the first voice signal, and the first voice signal is used to characterize the voice signal of the first interactive object at the current moment during the voice interaction process;

[0127] Step S1004: Discretize and encode the first speech signal to obtain a first speech feature sequence of the first speech signal, wherein different features in the first speech feature sequence are used to characterize different signal elements in the first speech signal;

[0128] Step S1006: Based on the first speech feature sequence, the historical speech feature sequence and the historical predicted feature sequence, a second speech feature sequence is predicted. The historical speech feature sequence is used to characterize the speech feature sequence corresponding to the speech signal sent by the first interactive object at a historical moment, and the historical predicted feature sequence is used to characterize the speech feature sequence predicted based on the historical speech feature sequence.

[0129] Step S1008: Decode the second speech feature sequence to obtain the second speech signal of the second interactive object at the current moment, wherein the second speech signal is used to characterize the speech signal of the second interactive object at the current moment during the speech interaction process;

[0130] Step S1010: Output the second voice signal by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the second voice signal.

[0131] The execution entity of the above-mentioned voice processing method is the server, and the first interface in the above steps is an interface used to obtain the first voice signal. The first interface includes a first parameter, and the value of the first parameter can be the first voice signal.

[0132] The second interface in the above steps is an interface for outputting a second voice signal. The second interface includes a second parameter, the value of which can be the second voice signal.

[0133] The preferred embodiments involved in the above-described embodiments are the same as the solutions, application scenarios, and implementation processes provided in the above-described embodiments, but are not limited to the solutions provided in the above-described embodiments.

[0134] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0135] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solutions of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0137] According to embodiments of this disclosure, a voice dialogue system is also provided, wherein the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be executed in a different order than that shown here.

[0138] Figure 11 is a schematic diagram of a voice dialogue system according to an embodiment of the present disclosure. As shown in Figure 11, the voice dialogue system includes:

[0139] The sound acquisition and encoding module 110 is configured to acquire the user's speech signal at the current moment and perform discretization encoding processing on the user's speech signal to obtain the first speech feature sequence of the user's speech signal, wherein different features in the first speech feature sequence are used to characterize different signal elements in the user's speech signal.

[0140] The multimodal processing module 112 is configured to predict the response speech signal at the current moment based on the first speech feature sequence, the historical speech feature sequence, and the historical prediction feature sequence to obtain the second speech feature sequence. The different features in the historical speech feature sequence are used to characterize the different signal elements in the historical user speech signal collected at the historical moment, and the historical prediction feature sequence is used to characterize the feature sequence obtained by predicting the historical response speech signal at the historical moment based on the historical speech feature sequence.

[0141] The voice decoding and playback module 114 is configured to perform discretization decoding on the second voice feature sequence to obtain the response voice signal at the current moment, and then play the response voice signal.

[0142] The sound acquisition and discretization encoding module 110, the multimodal processing module 112, and the speech discretization decoding and playback module 114 mentioned above correspond to steps S902 to S910 in the above embodiments. The two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in the above embodiments. The above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the AR / VR device provided in the above embodiments.

[0143] According to embodiments of this disclosure, a voice dialogue system is also provided, wherein the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be executed in a different order than that shown here.

[0144] Figure 12 is a schematic diagram of a voice dialogue system according to an embodiment of the present disclosure. As shown in Figure 12, the voice dialogue system includes:

[0145] The acquisition device 120 is configured to acquire the user's voice signal at the current moment;

[0146] The speech coding module 122 is configured to perform discretization coding processing on the user's speech signal to obtain a first speech feature sequence of the user's speech signal, wherein different features in the first speech feature sequence are used to characterize different signal elements in the user's speech signal.

[0147] The multimodal processing module 124 is configured to predict the response speech signal at the current moment based on the first speech feature sequence, the historical speech feature sequence, and the historical prediction feature sequence to obtain the second speech feature sequence. The different features in the historical speech feature sequence are used to characterize the different signal elements in the historical user speech signal collected at the historical moment, and the historical prediction feature sequence is used to characterize the feature sequence obtained by predicting the historical response speech signal at the historical moment based on the historical speech feature sequence.

[0148] The speech decoding module 126 is configured to perform discretization decoding on the second speech feature sequence to obtain the response speech signal at the current moment;

[0149] The playback device 128 is configured to play a response voice signal.

[0150] The aforementioned data acquisition device can be a microphone.

[0151] The aforementioned playback device can be a speaker, headphones, etc.

[0152] The acquisition device 120, speech discretization encoding module 122, multimodal processing module 124, speech discretization decoding and playback module 126, and playback device 128 mentioned above correspond to steps S902 to S910 in the above embodiments. The two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in the above embodiments. The above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and run in the AR / VR device provided in the above embodiments.

[0153] The preferred embodiments involved in the above-described embodiments are the same as the solutions, application scenarios, and implementation processes provided in the above-described embodiments, but are not limited to the solutions provided in the above-described embodiments.

[0154] Embodiments of this disclosure may provide a computer terminal, which may include a server and a client. The server may be any one of the servers in a server device group or a cloud server.

[0155] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0156] In this embodiment, the computer terminal described above can execute the program code in the method.

[0157] Optionally, FIG13 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure. As shown in FIG13, the computer terminal A may include: one or more (one shown in the figure) processors 1302, memory 1304, memory controller, and peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0158] The memory can be configured to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to computer terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0159] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: acquiring the first voice signal of the first interactive object at the current moment; encoding the first voice signal to obtain a first voice feature sequence; predicting a second voice feature sequence based on the first voice feature sequence, historical voice feature sequences, and historical prediction feature sequences, wherein the historical voice feature sequence is used to characterize the voice feature sequence corresponding to the voice signal sent by the first interactive object at a historical moment, and the historical prediction feature sequence is used to characterize the voice feature sequence predicted based on the historical voice feature sequence; decoding the second voice feature sequence to obtain the second voice signal of the second interactive object at the current moment; and outputting the second voice signal to the interactive device of the first interactive object.

[0160] Optionally, the processor may also execute program code for the following steps: predicting a second speech feature sequence based on a first speech feature sequence, a historical speech feature sequence, and a historical predicted feature sequence, including: inputting the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence into a multimodal processing model, and using the multimodal processing model to predict the second speech feature sequence.

[0161] Optionally, the processor may also execute program code for the following steps: the historical predicted feature sequence contains feature sequences of at least one type; inputting the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence into a multimodal processing model, including: inputting the first speech feature sequence and the historical speech feature sequence into a first channel of the multimodal processing model, wherein the first channel is used to input external speech feature sequences into the multimodal processing model; inputting feature sequences of at least one type into at least one second channel of the multimodal processing model, wherein the at least one second channel corresponds one-to-one with at least one type, wherein the second channel is used to input feature sequences generated by the multimodal processing model into the multimodal processing model.

[0162] Optionally, the processor may also execute program code that includes the following steps: at least one type is determined based on one of the following information: network environment data of the voice interaction process, output requirements carried in the first voice signal, and output requirements received before the start of the voice interaction process, wherein the output requirements are used to characterize the type of data that the second interaction object needs to output at the current moment.

[0163] Optionally, the processor may also execute program code that includes the following steps: the historical prediction feature sequence includes at least a predicted speech feature sequence, wherein different features in the predicted speech feature sequence are used to characterize different signal elements in the speech signal of the second interactive object at a historical moment.

[0164] Optionally, the processor may also execute program code for the following steps: the historical prediction feature sequence further includes a prediction text feature sequence, where different features in the prediction text feature sequence are used to characterize different text elements of the historical prediction text, and the historical prediction text is used to characterize the text corresponding to the speech signal of the second interactive object at a historical moment; the method further includes: decoding the text feature sequence output by the multimodal processing model to obtain the target text, wherein the target text is used to characterize the text corresponding to the speech signal of the second interactive object at the current moment; and outputting the target text to the interactive device of the first interactive object.

[0165] Optionally, the processor may also execute program code for the following steps: encoding a first speech signal to obtain a first speech feature sequence, including: performing discretization encoding processing on the first speech signal to obtain a first speech feature sequence, wherein different features in the first speech feature sequence are used to characterize different signal elements in the first speech signal; decoding a second speech feature sequence to obtain a second speech signal of the second interactive object at the current time, including: performing discretization decoding processing on the second speech feature sequence to obtain a second speech signal, wherein different features in the first speech feature sequence are used to characterize different signal elements in the first speech signal.

[0166] Optionally, the processor may also execute program code for the following steps: performing discretization encoding on the first speech signal to obtain a first speech feature sequence, including: using an encoder to extract features from the first speech signal to obtain a first speech vector corresponding to the first speech signal; using a quantizer to quantize the first speech vector through multiple preset features to obtain a first speech feature sequence; and performing discretization decoding on the second speech feature sequence to obtain a second speech signal, including: using a decoder to decode the second speech feature sequence to obtain a second speech signal.

[0167] In this embodiment, a first voice signal of the first interactive object at the current moment is acquired; the first voice signal is encoded to obtain a first voice feature sequence; a second voice feature sequence is predicted based on the first voice feature sequence, historical voice feature sequences, and historical predicted feature sequences; the second voice feature sequence is decoded to obtain a second voice signal of the second interactive object at the current moment; and the second voice signal is output to the interactive device of the first interactive object. By encoding and decoding the voice signal, the conversion between the voice signal and the voice feature sequence is realized, enabling a single node to predict the voice signal based on multiple feature sequences such as the first voice feature sequence, historical voice feature sequences, and historical predicted feature sequences. This avoids the delay introduced by forwarding the voice feature sequence between multiple nodes, and thus avoids the error superposition caused by data processing by multiple nodes. This achieves the technical effect of reducing voice interaction latency and improving voice interaction accuracy, thereby solving the technical problems of high latency and low accuracy in commonly used voice interaction schemes.

[0168] Those skilled in the art will understand that the structure shown in the figure above is illustrative, and computer terminal A can also be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, PDA, mobile internet device (MID), PAD, and other terminal devices. For example, computer terminal A may also include more or fewer components (such as network interface, display device, etc.) than shown in the figure above, or have a different configuration than shown in the figure above.

[0169] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0170] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0171] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal cluster, or in any mobile terminal in a mobile terminal cluster.

[0172] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring a first voice signal of a first interactive object at the current time; encoding the first voice signal to obtain a first voice feature sequence; predicting a second voice feature sequence based on the first voice feature sequence, a historical voice feature sequence, and a historical predicted feature sequence, wherein the historical voice feature sequence is used to characterize the voice feature sequence corresponding to the voice signal sent by the first interactive object at a historical time, and the historical predicted feature sequence is used to characterize the voice feature sequence predicted based on the historical voice feature sequence; decoding the second voice feature sequence to obtain a second voice signal of the second interactive object at the current time; and outputting the second voice signal to the interactive device of the first interactive object.

[0173] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: predicting a second speech feature sequence based on a first speech feature sequence, a historical speech feature sequence, and a historical predicted feature sequence, including: inputting the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence into a multimodal processing model, and using the multimodal processing model to predict the second speech feature sequence.

[0174] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: the historical predicted feature sequence includes a feature sequence of at least one type; inputting the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence into a multimodal processing model, including: inputting the first speech feature sequence and the historical speech feature sequence into a first channel of the multimodal processing model, wherein the first channel is used to input external speech feature sequences into the multimodal processing model; inputting feature sequences of at least one type into at least one second channel of the multimodal processing model, wherein the at least one second channel corresponds one-to-one with at least one type, wherein the second channel is used to input feature sequences generated by the multimodal processing model into the multimodal processing model.

[0175] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: at least one type is determined based on one of the following information: network environment data of the voice interaction process, output requirements carried in the first voice signal, and output requirements received before the start of the voice interaction process, wherein the output requirements are used to characterize the type of data that the second interactive object needs to output at the current moment.

[0176] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: the historical prediction feature sequence includes at least a predicted speech feature sequence, wherein different features in the predicted speech feature sequence are used to characterize different signal elements in the speech signal of the second interactive object at a historical moment.

[0177] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: the historical prediction feature sequence further includes a prediction text feature sequence, wherein different features in the prediction text feature sequence are used to characterize different text elements of the historical prediction text, and the historical prediction text is used to characterize the text corresponding to the speech signal of the second interactive object at a historical moment; the method further includes: decoding the text feature sequence output by the multimodal processing model to obtain target text, wherein the target text is used to characterize the text corresponding to the speech signal of the second interactive object at the current moment; and outputting the target text to the interactive device of the first interactive object.

[0178] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: encoding a first speech signal to obtain a first speech feature sequence, including: performing discretization encoding processing on the first speech signal to obtain the first speech feature sequence, wherein different features in the first speech feature sequence are used to characterize different signal elements in the first speech signal; decoding a second speech feature sequence to obtain a second speech signal of the second interactive object at the current time, including: performing discretization decoding processing on the second speech feature sequence to obtain the second speech signal, wherein different features in the first speech feature sequence are used to characterize different signal elements in the first speech signal.

[0179] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: discretizing and encoding a first speech signal to obtain a first speech feature sequence, including: extracting features from the first speech signal using an encoder to obtain a first speech vector corresponding to the first speech signal; quantizing the first speech vector using a quantizer through multiple preset features to obtain a first speech feature sequence; and discretizing and decoding a second speech feature sequence to obtain a second speech signal, including: decoding the second speech feature sequence using a decoder to obtain a second speech signal.

[0180] Embodiments of this disclosure also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0181] Embodiments of this disclosure also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0182] Embodiments of this disclosure also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0183] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0184] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are illustrative; for example, the division of units is a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0186] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0188] The above description represents the preferred embodiments of this disclosure. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure. Industrial applicability

[0189] By encoding and decoding the speech signal, the conversion between the speech signal and the speech feature sequence is realized. This allows a single node to predict the speech signal based on multiple feature sequences, such as the first speech feature sequence, historical speech feature sequences, and historical predicted feature sequences. This avoids the delay introduced by forwarding the speech feature sequence between multiple nodes, and thus avoids the accumulation of errors caused by data processing by multiple nodes. This achieves the technical effect of reducing the delay of voice interaction and improving the accuracy of voice interaction, thereby solving the technical problems of high latency and low accuracy in commonly used voice interaction schemes.

Claims

1. A speech processing method, comprising: Obtain the first voice signal of the first interactive object at the current moment; The first speech signal is encoded to obtain a first speech feature sequence; Based on the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence, a second speech feature sequence is predicted, wherein the historical speech feature sequence is used to characterize the speech feature sequence corresponding to the speech signal sent by the first interactive object at a historical moment, and the historical predicted feature sequence is used to characterize the speech feature sequence predicted based on the historical speech feature sequence. Decode the second speech feature sequence to obtain the second speech signal of the second interactive object at the current time; The second voice signal is output to the interactive device of the first interactive object.

2. The method of claim 1, wherein, The step of predicting a second speech feature sequence based on the first speech feature sequence, historical speech feature sequence, and historical predicted feature sequence includes: The first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence are input into a multimodal processing model, and the second speech feature sequence is predicted using the multimodal processing model.

3. The method of claim 2, wherein, The historical predicted feature sequence contains at least one type of feature sequence; the step of inputting the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence into the multimodal processing model includes: The first speech feature sequence and the historical speech feature sequence are input into the first channel of the multimodal processing model, wherein the first channel is used to input external speech feature sequences into the multimodal processing model; The feature sequences of the at least one type are respectively input into at least one second channel of the multimodal processing model, wherein the at least one second channel corresponds one-to-one with the at least one type, and the second channel is used to input the feature sequences generated by the multimodal processing model into the multimodal processing model.

4. The method according to claim 3, wherein, The at least one type is determined based on one of the following: network environment data of the voice interaction process, output requirements carried in the first voice signal, and output requirements received before the start of the voice interaction process, wherein the output requirements are used to characterize the type of data that the second interaction object needs to output at the current moment.

5. The method according to claim 3 or 4, wherein, The historical predicted feature sequence includes at least a predicted speech feature sequence, wherein different features in the predicted speech feature sequence are used to characterize different signal elements in the speech signal of the second interactive object at the historical moment.

6. The method according to claim 5, wherein, The historical prediction feature sequence also includes a prediction text feature sequence, wherein different features in the prediction text feature sequence are used to characterize different text elements of the historical prediction text, and the historical prediction text is used to characterize the text corresponding to the speech signal of the second interactive object at the historical moment; The method further includes: decoding the text feature sequence output by the multimodal processing model to obtain target text, wherein the target text is used to characterize the text corresponding to the speech signal of the second interactive object at the current time; Output the target text to the interaction device of the first interactive object.

7. The method according to any one of claims 1 to 6, wherein, Encoding the first speech signal to obtain a first speech feature sequence includes: performing discretization encoding processing on the first speech signal to obtain the first speech feature sequence, wherein different features in the first speech feature sequence are used to characterize different signal elements in the first speech signal; Decoding the second speech feature sequence to obtain the second speech signal of the second interactive object at the current time includes: performing discretization decoding processing on the second speech feature sequence to obtain the second speech signal, wherein different features in the first speech feature sequence are used to characterize different signal elements in the first speech signal.

8. The method according to claim 7, wherein, The step of discretizing and encoding the first speech signal to obtain the first speech feature sequence includes: using an encoder to extract features from the first speech signal to obtain a first speech vector corresponding to the first speech signal; and using a quantizer to quantize the first speech vector through multiple preset features to obtain the first speech feature sequence. The step of discretizing and decoding the second speech feature sequence to obtain the second speech signal includes: decoding the second speech feature sequence using a decoder to obtain the second speech signal.

9. A voice dialogue method, comprising: Collect the user's voice signal at the current moment; The user's voice signal is encoded to obtain a first voice feature sequence; Based on the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence, a second speech feature sequence is predicted, wherein the historical speech feature sequence is used to characterize the speech feature sequence corresponding to the user speech signal collected at a historical time, and the historical predicted feature sequence is used to characterize the speech feature sequence predicted based on the historical speech feature sequence. The second speech feature sequence is decoded to obtain the response speech signal at the current moment; Play the response voice signal.

10. A speech processing method, comprising: The first voice signal of the first interactive object at the current moment is obtained by calling the first interface, wherein the first interface includes a first parameter and the parameter value of the first parameter includes the first voice signal. The first speech signal is encoded to obtain a first speech feature sequence; Based on the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence, a second speech feature sequence is predicted, wherein the historical speech feature sequence is used to characterize the speech feature sequence corresponding to the speech signal sent by the first interactive object at a historical moment, and the historical predicted feature sequence is used to characterize the speech feature sequence predicted based on the historical speech feature sequence. Decode the second speech feature sequence to obtain the second speech signal of the second interactive object at the current time; The second voice signal is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the second voice signal.

11. A voice dialogue system, comprising: The sound acquisition and encoding module is configured to acquire the user's speech signal at the current moment and encode the user's speech signal to obtain a first speech feature sequence; The multimodal processing module is configured to predict a second speech feature sequence based on the first speech feature sequence, the historical speech feature sequence, and the historical prediction feature sequence. The historical speech feature sequence is used to characterize the speech feature sequence corresponding to the speech signal sent by the first interactive object at a historical moment, and the historical prediction feature sequence is used to characterize the speech feature sequence predicted based on the historical speech feature sequence. The voice decoding and playback module is configured to decode the second voice feature sequence to obtain the response voice signal at the current moment, and play the response voice signal.

12. A voice dialogue system, comprising: The data acquisition device is configured to acquire the user's voice signal at the current moment; The speech encoding module is configured to encode the user's speech signal to obtain a first speech feature sequence; The multimodal processing module is configured to predict a second speech feature sequence based on the first speech feature sequence, the historical speech feature sequence, and the historical prediction feature sequence. The historical speech feature sequence is used to characterize the speech feature sequence corresponding to the speech signal sent by the first interactive object at a historical moment, and the historical prediction feature sequence is used to characterize the speech feature sequence predicted based on the historical speech feature sequence. The voice decoding module is configured to decode the second voice feature sequence to obtain the response voice signal at the current moment; The playback device is configured to play the response voice signal.

13. A computer terminal, comprising: Memory, which stores executable programs; A processor is configured to run the program, wherein the program executes the following method during runtime: acquiring a first voice signal of a first interactive object at the current time; encoding the first voice signal to obtain a first voice feature sequence; predicting a second voice feature sequence based on the first voice feature sequence, a historical voice feature sequence, and a historical predicted feature sequence, wherein the historical voice feature sequence is used to characterize the voice feature sequence corresponding to a voice signal sent by the first interactive object at a historical time, and the historical predicted feature sequence is used to characterize the voice feature sequence predicted based on the historical voice feature sequence; decoding the second voice feature sequence to obtain a second voice signal of a second interactive object at the current time; and outputting the second voice signal to the interactive device of the first interactive object.

14. The computer terminal according to claim 13, wherein, The processor is also configured to run a program corresponding to the following method: inputting the first speech feature sequence, the historical speech feature sequence, and the historical predicted feature sequence into a multimodal processing model, and using the multimodal processing model to predict the second speech feature sequence.

15. The computer terminal according to claim 14, wherein, The historical predicted feature sequence includes at least one type of feature sequence; the processor is further configured to run a program corresponding to the following method: inputting the first speech feature sequence and the historical speech feature sequence into a first channel of the multimodal processing model, wherein the first channel is used to input external speech feature sequences into the multimodal processing model; inputting the at least one type of feature sequence into at least one second channel of the multimodal processing model, wherein the at least one second channel corresponds one-to-one with the at least one type, wherein the second channel is used to input the feature sequence generated by the multimodal processing model into the multimodal processing model.

16. The computer terminal according to claim 15, wherein, The at least one type is determined based on one of the following: network environment data of the voice interaction process, output requirements carried in the first voice signal, and output requirements received before the start of the voice interaction process, wherein the output requirements are used to characterize the type of data that the second interaction object needs to output at the current moment.

17. The computer terminal according to claim 15 or 16, wherein, The historical predicted feature sequence includes at least a predicted speech feature sequence, wherein different features in the predicted speech feature sequence are used to characterize different signal elements in the speech signal of the second interactive object at the historical moment.

18. The computer terminal according to claim 17, wherein, The historical prediction feature sequence further includes a prediction text feature sequence, wherein different features in the prediction text feature sequence are used to characterize different text elements of the historical prediction text, and the historical prediction text is used to characterize the text corresponding to the speech signal of the second interactive object at the historical moment; the processor is also configured to run a program corresponding to the following method: decoding the text feature sequence output by the multimodal processing model to obtain target text, wherein the target text is used to characterize the text corresponding to the speech signal of the second interactive object at the current moment; and outputting the target text to the interactive device of the first interactive object.

19. A computer-readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the method according to any one of claims 1 to 10.

20. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

Images