Method and apparatus for voice processing, device, storage medium, and program product

Abstract

A method and apparatus for voice processing, a device, a storage medium, and a program product. The method comprises: acquiring a response text for a question voice of a target user (410); at least on the basis of a speech conversion requirement of the response text, selecting at least one of a first speech synthesis model on a local device and a second speech synthesis model on a server for executing a speech synthesis function on the response text (420); and acquiring a response voice corresponding to the response text, the response voice being generated after the speech synthesis function is executed on the response text by using the selected at least one speech synthesis model (430).

Description

Method, device, equipment, storage medium and program product for speech processing

[0001] The present application claims priority to the Chinese patent application No. 202411412273.7, filed on October 10, 2024, entitled “Method, device, equipment, storage medium and program product for speech processing”, the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The example embodiments of the present disclosure generally relate to the field of computer, and in particular, to a method, device, electronic equipment, computer readable storage medium and computer program product for speech processing. BACKGROUND

[0003] With the development of Internet technology, more and more applications or platforms, etc. provide speech processing functions, which bring great convenience to the general public. Speech processing functions include speech synthesis (TTS, Text to Speech) function (also can be referred to as text to speech function), question and answer function, speech recognition (ASR, Automatic Speech Recognition) function (also can be referred to as speech to text function), etc. The application or platform with speech processing function can provide speech services relying on speech processing function to the user by using the trained machine learning model. SUMMARY

[0004] In a first aspect of the present disclosure, a speech processing method is provided. The method is applied to a client, and the method comprises: obtaining a response text of a question voice of a target user; selecting at least one speech synthesis model of a first speech synthesis model locally and a second speech synthesis model at a server based on at least a speech conversion requirement of the response text, for performing a speech synthesis function on the response text; and obtaining a response voice corresponding to the response text, the response voice being generated by performing the speech synthesis function on the response text by using the selected at least one speech synthesis model.

[0005] In a second aspect of the present disclosure, a device for speech processing is provided. The device is applied to a client, and the device comprises: a response text obtaining module configured to obtain a response text of a question voice of a target user; a synthesis model selecting module configured to select at least one speech synthesis model of a first speech synthesis model locally and a second speech synthesis model at a server based on at least a speech conversion requirement of the response text, for performing a speech synthesis function on the response text; and a response voice obtaining module configured to obtain a response voice corresponding to the response text, the response voice being generated by performing the speech synthesis function on the response text by using the selected at least one speech synthesis model.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions, wherein when executed by a processor, the computer-executable instructions implement the method according to a first aspect of this disclosure.

[0009] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 illustrates a schematic diagram of an example architecture for information processing according to some embodiments of the present disclosure;

[0013] Figure 3 illustrates a schematic diagram of an example of information processing according to some embodiments of the present disclosure;

[0014] Figure 4 shows a flowchart of a method for speech processing according to some embodiments of the present disclosure;

[0015] Figure 5 shows an example structural block diagram of a speech processing apparatus according to some embodiments of the present disclosure; and

[0016] Figure 6 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0023] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0025] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0026] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0027] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as an input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0028] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In this example environment 100, an application 112 is installed on a client 110. A user 140 can interact with the application 112 via the client 110 and / or an attached device of the client 110. For example, the application 112 can capture the voice 145 of the user 140 via a voice capture device (e.g., a microphone) of the client 110.

[0029] In embodiments of this disclosure, application 112 can be any suitable application with voice processing capabilities. For example, application 112 can be a social application, a chat application, a media application, and so on. Application 112 can, for example, provide a digital assistant for human-computer dialogue. The digital assistant supports text dialogue services, voice dialogue services, and content dialogue in other modalities with user 140. In some embodiments, application 112 or the digital assistant therein can utilize machine learning models. For example, application 112 or the digital assistant therein can utilize machine learning models to provide question-and-answer services to user 140. The digital assistant's response to the user can be determined based on the model output of the machine learning model.

[0030] This machine learning model can be deployed locally on client 110 (e.g., machine learning model 114) or deployed on other devices (e.g., machine learning model 130 on server 120). A machine learning model deployed locally on client 110, such as machine learning model 114, can be referred to as an offline machine learning model. A machine learning model deployed on other devices, such as machine learning model 130, can be referred to as an online machine learning model.

[0031] Both machine learning model 114 and machine learning model 130 can be based on any suitable model architecture, including but not limited to Transformer models, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep neural networks (DNNs), and so on. In some embodiments, machine learning model 114 and / or machine learning model 130 can be based on a language model (LM). A language model, by learning from a large corpus, is capable of question answering.

[0032] In some embodiments, machine learning model 114 and / or machine learning model 130 may be content-generating models capable of generating corresponding outputs based on model inputs. In some embodiments, language-based machine learning models are capable of receiving text-modal model inputs (e.g., natural language and / or machine language) and / or non-text-modal model inputs (e.g., images, speech, video, etc.), and are capable of generating desired outputs based on model inputs and prompts. The prompts are used to guide the machine learning model to generate model outputs that address the user's needs indicated by the model inputs. In applications supporting user dialogue, user 140's input may be provided to machine learning model 114 and / or machine learning model 130 as at least a portion of the model inputs (other portions may include prompts). This user input is considered a question. Based on the model outputs, corresponding responses can be generated and provided to user 140.

[0033] It is important to note that both machine learning model 114 and machine learning model 130 may include one or more machine learning models. If multiple machine learning models are included, their functions, structures, and uses may be the same or different. In some embodiments, machine learning model 114 and / or machine learning model 130 may include at least several speech-related machine learning models, such as a machine learning model for performing text-to-speech (TTS) synthesis (which may be simply referred to as a TTS model), a machine learning model for performing automatic speech recognition (ASR) (which may be simply referred to as an ASR model), and a machine learning model for performing question answering (which may be simply referred to as a question answering model). The input of the ASR model is speech, and the output is text. The input of the TTS model is text, and the output is the corresponding speech. The input of the question answering model is the question text, and the output is the corresponding response text.

[0034] In environment 100, if application 112 is active, client 110 can present the user interface 150 of application 112. User interface 150 may include various pages that application 112 can provide, such as a user-digital assistant dialogue page (where the current and historical dialogues, including text dialogue content, may be presented), and so on. In some embodiments, client 110 may play voice via user interface 150, which may include a question voice 152 from the user and a response voice 154 to the question voice 152.

[0035] In some embodiments, client 110 communicates with server 120 to provide services to application 112. For example, server 120 may invoke machine learning model 130 to support human-computer dialogue between application 112 and user 140 based on the output of machine learning model 130.

[0036] Client 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, client 110 may also support any type of user-facing interface (such as "wearable" circuitry).

[0037] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 120 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 120 may be implemented based on a cloud environment.

[0038] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0039] As mentioned earlier, applications or platforms with speech processing capabilities can use trained machine learning models to provide users with voice services that rely on speech processing functionality. Traditionally, machine learning models are typically deployed on the server side; that is, applications or platforms with speech processing capabilities use server-side machine learning models to provide voice services to users. The process of an application or platform calling the machine learning model deployed on the server side can be affected by network communication. In cases of poor network communication, the client may be unable to call the machine learning model deployed on the server side.

[0040] In view of the above, according to embodiments of the present disclosure, an improved speech processing scheme is provided. According to the scheme of the embodiments of the present disclosure, a response text to a question posed by a target user is obtained. At least one speech synthesis model, selected from a local first speech synthesis model and a server-side second speech synthesis model, is used to perform speech synthesis on the response text, based at least on the speech conversion requirements of the response text. The response speech corresponding to the response text is obtained; the response speech is generated after performing speech synthesis on the response text using the selected at least one speech synthesis model.

[0041] In this way, the corresponding voice response can be determined by combining network communication capabilities and the requirements for voice conversion of the response text. This can meet the TTS needs of users under any network communication and different voice conversion requirements, and improve the user experience while ensuring accuracy.

[0042] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0043] Figure 2 illustrates a schematic diagram of an example architecture 200 for voice processing according to some embodiments of the present disclosure. Architecture 200 can be implemented at a client 110. For ease of discussion, architecture 200 will be described with reference to the environment 100 of Figure 1. It should be noted that the operations performed by the aforementioned client 110, as well as the operations performed by the client 110 described below, can specifically be performed by a related application (e.g., application 112) installed on the client 110. In some embodiments, the operations performed on the client 110 can be completed with the assistance of the server 120.

[0044] As shown in Figure 2, architecture 200 involves client 110 and server 120. Client 110 may include strategy module 210, online request module 220, and offline invocation module 230. Offline invocation module 230 is used to interact with local model capability 240, which can provide machine learning models deployed locally on client 110 (e.g., may include at least ASR model 241, question answering model 242, and TTS model 243). ASR model 241 may also be referred to as a first speech recognition model, question answering model 242 may also be referred to as a first question answering model, and TTS model 243 may also be referred to as a first speech synthesis model.

[0045] The online request module 220 interacts with the server 120 to invoke the machine learning model deployed on the server 120. The server 120 may include an access layer 250, a voice module 260, and a question-answering module 270. The server 120 interacts with the client 110 via the access layer 250. The voice module 260 interacts with the voice service 280, which can provide a voice processing model deployed on the server 120 (e.g., it may include an ASR model 281 and a TTS model 282). The ASR model 281 may also be referred to as a second speech recognition model, and the TTS model 282 may also be referred to as a second speech synthesis model. The question-answering module 270 interacts with the question-answering model 290. The question-answering model 290 may also be referred to as a second question-answering model.

[0046] During voice processing, client 110 can receive voice questions from users (e.g., user 140). These voice questions may be received during a question-and-answer dialogue. It is understood that the voice in this disclosure can be any suitable voice of any duration, language, tone, etc. A question-and-answer dialogue may be, for example, a conversation between a user and a digital assistant, and the dialogue can occur in any dialogue window.

[0047] The strategy module 210 can determine multiple strategies for performing speech processing on the question speech in response to determining that the client 110 has received the question speech. These multiple strategies may include, for example, an ASR distribution strategy, a response strategy, and a TTS distribution strategy. As an example, the ASR distribution strategy can be used to determine which machine learning model to select to perform ASR functionality on the question speech. The question-answering strategy can be used to determine which machine learning model to select to process the question text. The TTS distribution strategy can be used to determine which machine learning model to select to perform TTS functionality on the response text.

[0048] Specifically, the policy module 210 can detect the network communication capability between the client 110 and the server 120, and determine the ASR distribution policy based at least on the network communication capability (e.g., it can also be based on user requirements specifying the machine learning model to perform the ASR function, the version of the machine learning model, its source, etc.). The network communication capability can be determined based on one or more factors such as network signal strength, network connection speed, and network latency, and can be used to measure the network connectivity between the client 110 and the server 120. The policy module 210 can detect the network communication capability between the client 110 and the server 120 using a network probe.

[0049] The strategy module 210 can also determine the capability level corresponding to the network communication capability based on the detection results. For example, the strategy module 210 can predetermine the correspondence between network communication capabilities and capability levels. For instance, the strategy module 210 can determine that when the network communication capability is in the interval [a1, b1), it is at capability level A; when it is in the interval [a2, b2), it is at capability level B; when it is in the interval [a3, b3), it is at capability level C, and so on. It is understood that there are no identical values ​​in different intervals. For example, there are no identical values ​​in the intervals [a1, b1) and [a2, b2).

[0050] The strategy module 210 can determine the ASR distribution strategy for problematic speech based on the network communication capabilities between the client 110 and the server 120, and select the appropriate ASR model to perform ASR functions on the problematic speech according to the determined ASR distribution strategy. In some embodiments, the strategy module 210 can determine that there is currently no network communication or the current network communication is poor, and it is impossible to perform ASR functions using the machine learning model deployed on the server 120, in response to the network communication capability being lower than the third capability level. In this case, the strategy module 210 selects the local ASR model 241 to perform ASR functions on the problematic speech (this situation can also be referred to as offline ASR function). It should be understood that the third capability level here can be any appropriate network capability level configured according to actual needs to limit the network conditions suitable for calling offline ASR functions.

[0051] Strategy module 210 can instruct offline invocation module 230 to perform ASR functionality on the question speech using ASR model 241 in local model capability 240. Offline invocation module 230, for example, can respond to an instruction received from strategy module 210 by determining the model input for ASR model 241 based on the question speech. Offline invocation module 230 can provide the model input to ASR model 241 and obtain the model output of ASR model 241 for this model input, which can indicate the ASR text for the question speech (also referred to as offline ASR text). Client 110 can determine the received offline ASR text as the question text for the question speech. The question text determined by ASR model 241 can be referred to as the offline question text.

[0052] In some embodiments, policy module 210 may determine that network communication is currently present or that the current network communication is good in response to network communication capabilities exceeding the fourth capability level. In this case, policy module 210 may select ASR model 241 and / or ASR model 281 to perform ASR functionality on the problematic speech. In this case, policy module 210 may instruct offline invocation module 230 to perform offline ASR functionality on the problematic speech using ASR model 241 in local model capability 240, and may instruct online request module 220 to send the problematic speech to server 120 to request ASR functionality to be performed on the problematic speech using ASR model 281 (this situation may also be referred to as online ASR functionality). It should be understood that the fourth capability level here can also be any appropriate network capability level configured according to actual needs to define the network conditions suitable for invoking offline ASR functionality and online ASR functionality. The specific details of performing ASR functionality on the problematic speech using ASR model 241 in local model capability 240 will not be elaborated here.

[0053] Regarding the specific method of sending the question voice to the server 120, in some embodiments, a network connection can be established between the client 110 and the server 120. This network connection can be a long connection conforming to the Transmission Control Protocol (TCP). For example, the client 110 can establish a long connection with the network interface 255 of the access layer 250 of the server 120 through a three-way interaction process (also known as a three-way handshake) as specified by the TCP protocol. The server 120 can receive the question voice through the network interface 255 and provide it to the voice module 260. The voice module 260 can determine the model input for the ASR model 281 based on the question voice. The voice module 260 can provide this model input to the ASR model 281 at the voice service 280 and obtain the model output of the ASR model 281 for this model input. This model output can indicate the ASR text (also known as the online ASR text) for the question voice.

[0054] It's important to note that interactions between client 110 and server 120 typically involve some time consumption. Furthermore, the model size of the machine learning model deployed locally on client 110 is usually smaller than that deployed on server 120. The model size of each machine learning model is related to its parameter size, model structure complexity, and other factors. Generally, the larger the parameter size or the more complex the structure of a machine learning model, the larger its model size. Larger model sizes require greater resource overhead, including but not limited to computational resources, memory resources, and time consumption. In summary, even if client 110 simultaneously invokes ASR model 241 and sends a question voice to server 120, client 110 can still first obtain the model output of its local ASR model 241 (i.e., the offline ASR text, which in this case can also be referred to as the first candidate question text).

[0055] Client 110 (specifically, for example, strategy module 210) may, in response to obtaining the first candidate question text, determine a quality score for the first candidate question text. Strategy module 210 may determine the quality score of the first candidate question text using any suitable method. For example, strategy module 210 may determine the quality score of the first candidate question text based on a predetermined scoring rule. Alternatively, strategy module 210 may determine the quality score of the first candidate question text using a trained machine learning model (e.g., an evaluation model 244 deployed locally).

[0056] The strategy module 210 may also store predetermined quality requirements. Quality requirements may, for example, indicate a threshold score. In response to a first candidate question text meeting the quality requirements (e.g., reaching a threshold score), the strategy module 210 may identify the first candidate question text as the question text corresponding to the question speech. In this case, the strategy module 210 may determine that it is unnecessary to determine the question text via the online ASR function and may instruct the online request module 220 to send a stop generation instruction to the server 110 to stop speech synthesis of the question speech (i.e., stop the online ASR function).

[0057] The strategy module 210 can also delete the received first candidate question text if its quality score does not meet the quality requirements (e.g., the quality score does not reach the threshold score), and determine the online ASR text (which can also be called the second candidate question text) as the question text corresponding to the question speech if it receives an online ASR text from the server 120. The question text determined by the ASR model 281 can be called the online question text. Thus, by verifying the quality of the first candidate question text, it can be determined whether the online ASR function needs to continue, which helps to reduce resource consumption and improve the efficiency of determining the question text.

[0058] After determining the question text for the question speech, strategy module 210 can determine a question-and-answer strategy for the question text and, based on this strategy, determine the corresponding response text. For example, strategy module 210 can determine the question complexity of the question text and determine the question-and-answer strategy based on this complexity. Strategy module 210 can determine the question complexity of the question text in any appropriate manner. For example, strategy module 210 can determine the question complexity of the question text based on predetermined determination rules. Alternatively, strategy module 210 can use a trained machine learning model (e.g., an evaluation model 244 deployed locally) to determine the question complexity of the question text. Evaluation model 244 can include multiple machine learning models, where the machine learning model used to determine the question complexity of the question text and the machine learning model used to determine the quality score of the first candidate question text mentioned above can be different machine learning models.

[0059] The strategy module 210 can also store predetermined complexity requirements. If the complexity of the question text does not meet the complexity requirements, the strategy module 210 can determine that the question text is relatively simple and does not require the question-answering model 290 at the server 120 to determine the corresponding response text. In this case, the strategy module 210 can instruct the offline invocation module 230 to invoke the question-answering model 242 in the local model capability 240. For example, in response to receiving an instruction from the strategy module 210, the offline invocation module 230 can determine the model input for the question-answering model 242 based on the question text. The offline invocation module 230 can provide the model input to the question-answering model 242 and obtain the model output of the question-answering model 242 for this model input. This model output can indicate the response text for the question text (also referred to as the offline response text).

[0060] The strategy module 210 can also respond to situations where the complexity of the question text meets the complexity requirements, indicating that the question text is relatively complex and requires the use of the question-answering model 290 at the server 120 to determine the corresponding response text. In this case, the strategy module 210 can instruct the online request module 220 to send the question text to the server 120 to request the use of the question-answering model 290 to determine the corresponding response text based on the question text. The server 120 can receive the question text via the network interface 255 and provide it to the question-answering module 270.

[0061] The question-answering module 270 can determine the model input for the question-answering model 290 based on the question text. The question-answering module 270 can provide this model input to the question-answering model 290 and obtain the model output of the question-answering model 290 for this model input. This model output can indicate the response text (also referred to as the online response text) for the question text. It should be noted that, in some embodiments, to improve the efficiency of determining the online response text, when the question text is determined using the ASR model 281 at the server 120, the voice module 260 can, in response to obtaining the online ASR text, directly provide the online ASR text to the question-answering module 270 to instruct the question-answering module 270 to determine the corresponding online response text.

[0062] Regarding the specific method for determining the model input for the question-answering model (the local question-answering model 242 and the server-side question-answering model 290), in some embodiments, to improve the accuracy of the determined question-answer text, the user's contextual information can be provided to the question-answering model along with the question text. The user's contextual information may include one or more of the following: the user's historical interaction information (e.g., historical question voice, historical question text, historical response text, historical response voice), multimodal information from the user (e.g., images, audio, video, documents, etc. sent / uploaded by the user), the user's environmental information, the user's user attribute information, and the application's version information.

[0063] In summary, determining the response text based on the complexity of the question text can improve efficiency while ensuring accuracy.

[0064] The question-and-answer module 270 can send the response text to the network interface 255 of the access layer 250 (this process can also be referred to as the question-and-answer module 270 returning the response text to the network interface 255). The access layer 250 can send the obtained response text to the online request module 220 of the client 110. The policy module 210 can, in response to the client 110 obtaining the response text (i.e., obtaining the online response text or the offline response text), determine the TTS distribution policy for the response text, and select the appropriate TTS model to perform TTS functions on the response text according to the determined TTS distribution policy.

[0065] The strategy module 210 can determine a TTS distribution strategy for the response text based at least on the speech-to-text requirements of the response text. The strategy module 210 can determine the speech-to-text requirements of the response text in any suitable manner. For example, the strategy module 210 can determine the speech-to-text requirements of the response text based on predetermined rules or algorithms. For instance, the strategy module 210 can determine the corresponding speech-to-text requirements based on the number of text units included in the response text, at least one keyword included in the response text, etc. As another example, the strategy module 210 can also use a trained machine learning model (e.g., an evaluation model 244 deployed locally) to determine the speech-to-text requirements of the response text. It is understood that the evaluation model 244 can include multiple machine learning models, and the machine learning model used to determine the speech-to-text requirements of the response text, the machine learning model used to determine the quality score of the first candidate question text, the machine learning model used to determine the question complexity of the question text, etc., can be different machine learning models.

[0066] The speech conversion requirement can specify one or more of the following: intonation requirements for the response text, voice timbre requirements for the response text, and voice quality requirements for the response text. As an example, the TTS model 282 at server 120 will be larger than the local TTS model 243 at client 110. Compared to TTS model 243, TTS model 282 can generate a more natural (i.e., more human-like) tone, higher quality speech, or offer more timbre options. For instance, if the speech conversion requirement for the response text specifies generating a natural tone with a specific timbre, the strategy module 210 can determine, based on this speech conversion requirement, that the TTS function needs to be performed using the TTS model 282 at server 120 (this can also be referred to as an online TTS function). If the speech conversion requirement of the response text indicates a low requirement for speech quality, the strategy module 210 can determine that it is not necessary to use the TTS model 282 at the server 120 to perform the TTS function. The strategy module 210 can call the local TTS model 243 to perform the TTS function (this situation can also be called offline TTS function).

[0067] In some embodiments, the policy module 210 may also determine the TTS distribution policy based on the network communication capabilities between the client 110 and the server 120. The specific method for determining network communication capabilities will not be elaborated here. In some embodiments, the policy module 210 may determine that there is currently no network communication or the current network communication is poor, making it impossible to perform online TTS functionality using the TTS model 282 deployed on the server 120, in response to a network communication capability lower than a first capability level. In this case, the policy module 210 selects a local TTS model 243 to perform offline TTS functionality on the response text. It should be understood that the first capability level here can also be any appropriate network capability level configured according to actual needs to define the network conditions suitable for invoking offline TTS functionality.

[0068] Alternatively or additionally, in some embodiments, the policy module 210 may determine that network communication is currently present or that the current network communication is good in response to network communication capabilities exceeding a second capability level. It should be understood that the second capability level here can also be any appropriate network capability level configured according to actual needs to define network conditions suitable for invoking offline and online TTS functions. It should be noted that the first, second, third, and fourth capability levels mentioned above can be the same or different capability levels, and this disclosure does not limit this. In this case, the policy module 210 can determine that both online and offline TTS functions can be selected (i.e., TTS model 282 and / or TTS model 243 can be selected to perform TTS functions on the response text). The policy module 210 can further determine a TTS distribution strategy based on the speech conversion requirements of the response text. The specific methods for determining the TTS distribution strategy based on speech conversion requirements will not be elaborated here.

[0069] In some embodiments, if the TTS distribution strategy determines that TTS model 243 should be selected, the strategy module 210 can instruct the offline invocation module 230 to perform offline TTS functionality on the response text using the TTS model 243 in the local model capability 240. The offline invocation module 230, for example, can determine the model input for TTS model 243 based on the response text in response to an instruction received from the strategy module 210. The offline invocation module 230 can provide the model input to the TTS model 243 and obtain the model output of the TTS model 243 for this model input, which can indicate the TTS speech (also referred to as offline TTS speech) for the response text. The client 110 can determine the received offline TTS speech as the response speech for the response text. The response speech determined by the TTS model 243 can be referred to as the offline response speech, the first response speech, etc.

[0070] In some embodiments, if the TTS distribution strategy determines that TTS model 282 should be selected, the strategy module 210 can instruct the online request module 220 to send a response text to the server 120 to request the use of TTS model 282 to perform online TTS functionality on the response text. The server 120 can receive the response text via network interface 255 and provide it to the voice module 260. The voice module 260 can determine the model input for TTS model 282 based on the response text. The voice module 260 can provide this model input to the TTS model 282 at the voice service 280 and obtain the model output of TTS model 282 for this model input, which can indicate the TTS voice for the response text (also referred to as online TTS voice). The client 110 can determine the received online TTS voice as the response voice for the response text. The response voice determined by TTS model 282 can be referred to as online response voice, second response voice, etc.

[0071] Client 110 can provide a response voice to the user in response to receiving a response voice (offline response voice and / or online response voice). Client 110 can, for example, play the response voice in the user's interaction interface with the digital assistant. Client 110 can also play the response voice to the user via a speaker.

[0072] It is understandable that when client 110 simultaneously calls its local TTS model 243 and sends a response text to server 120, client 110 will first obtain the offline response voice and then the online response voice. Therefore, in some embodiments, to shorten user waiting time and improve the efficiency of user obtaining response voice, when TTS model 282 at server 120 is selected, client 110 can also simultaneously call its local TTS model 243 to determine the offline response voice corresponding to the response text (that is, client 110 can simultaneously select local TTS model 243 and server TTS model 282 at server 120).

[0073] Client 110 can respond to receiving offline response audio by playing at least a portion of it. Client 110 can then respond to receiving online response audio generated by TTS model 282 from server 120 by stopping playback of the offline response audio. Client 110 can determine the corresponding playback start point in the online response audio based on the playback stop point of the offline response audio, and start playing online response audio with better intonation and voice quality from that start point. For example, if client 110 receives online response audio at the 4th second of playing offline response audio, it can stop playing offline response audio and resume playing online response audio from the 4th second of online response audio.

[0074] In some embodiments, since the time difference between the offline and online response voices obtained by the client 110 is typically measured in milliseconds or seconds, when the TTS model 282 is selected, to further reduce the resource consumption required by the TTS function, the client 110 can determine the guiding text based on the content of the response text. For example, if the user's question indicates a movie recommendation, and the response text is a description of the movie, the guiding text may include the text "The movie I recommend is 'XXXX,' and below is a synopsis of the movie...". The client 110 may provide only a concise guiding text to the local TTS model 243 without providing the response text, and obtain the offline TTS voice (which can be referred to as the offline guiding voice) for the guiding text. The client device 110 may first play the short offline guiding voice to the user, and after obtaining the online response voice corresponding to the response text, directly play the online response voice after the offline guiding voice has finished playing.

[0075] Therefore, according to the embodiments of this disclosure, the method for determining the corresponding response speech can be determined by combining network communication capabilities, speech conversion requirements for response text, etc., which can meet the TTS needs of users under any network communication and different speech conversion requirements, and improve the user experience while ensuring accuracy.

[0076] Referring to FIG3, FIG3 illustrates a schematic diagram of an example 300 of information processing according to some embodiments of the present disclosure. In example 300, the ASR module 310 of client 110 can receive a question voice from user 140. The ASR module 310 can determine an ASR distribution strategy 312 for performing ASR functions on the question voice based on the question voice, the network communication capabilities between client 110 and server 120, etc. The ASR distribution strategy 312 can be used to determine at least one of the ASR model 241 local to client 110 and the ASR model 281 at server 120 to perform ASR functions on the question voice.

[0077] ASR module 310 can determine ASR request 314, which includes at least the problematic speech. If ASR model 241 can be selected based on ASR distribution strategy 312, ASR module 310 can directly provide ASR request 314 to the local ASR model 241 to perform offline ASR function on the problematic speech using the local ASR model 241. If ASR model 281 can be selected based on ASR distribution strategy 312, ASR module 310 can provide ASR request 314 to server 120 to request the use of ASR model 281 to perform online ASR function on the problematic speech.

[0078] Client 110 can obtain the question text obtained by performing ASR (Offline ASR and Online ASR) on the question speech. In some embodiments, client 110 can determine whether end model 320 (end model 320 corresponds to question-answering model 242 in FIG2) can determine the corresponding response text based on the question text. For example, client 110 can determine whether end model 320 can determine the corresponding response text based on whether end model 320 can generate the correct result for the question text. Client 110 can also determine whether end model 320 can determine the corresponding response text based on the question complexity of the question text (for example, if the question complexity is high, it can be determined that end model 320 cannot determine the corresponding response text based on the question text).

[0079] If the endpoint model 320 can determine the corresponding response text based on the question text, the client 110 can provide at least the question text (which may also include user input text from the user, such as text corresponding to the user's historical interactions, text provided by the user along with the question voice, etc.) to the endpoint model 320. The client 110 can directly obtain the response text from the endpoint model 320. If the endpoint model 320 cannot determine the corresponding response text based on the question text, the client 110 can at least send the question text to the server 120 to request the question-answering model 290 at the server 129 to determine the response text corresponding to the question text.

[0080] Furthermore, if the response text is determined using client model 320, client 110 can determine a TTS distribution strategy 330 for the response text based on the speech conversion requirements of the response text, the network communication capabilities between client 110 and server 120, etc. The TTS distribution strategy 330 can be used to determine at least one of the local TTS model 243 of client 110 and the TTS model 282 at server 120 to perform TTS functionality on the response text.

[0081] If the TTS distribution strategy 330 determines that TTS model 243 should be selected, client 110 can directly call the local TTS model 243 to perform offline TTS functionality on the response text. If the TTS distribution strategy 330 determines that TTS model 282 should be selected, client 110 can provide the response text to server 120 to request the use of TTS model 282 to perform online TTS functionality on the response text. Client 110 can obtain the response audio obtained from performing TTS functionality (offline TTS function and online TTS function) on the response text. For example, client 110 can play the response audio to user 140 via playback module 340.

[0082] Figure 4 shows a flowchart of a method 400 for speech processing according to some embodiments of the present disclosure. Method 400 can be implemented at client 110.

[0083] In box 410, client 110 obtains the text of the response to the question spoken by the target user.

[0084] In box 420, client 110 selects at least one of a local first speech synthesis model and a server-side second speech synthesis model, based at least on the speech conversion requirements of the response text, to perform speech synthesis on the response text.

[0085] In box 430, client 110 obtains the response speech corresponding to the response text. The response speech is generated after performing speech synthesis on the response text using at least one selected speech synthesis model.

[0086] In some embodiments, selecting at least one speech synthesis model from a local first speech synthesis model and a server-side second speech synthesis model includes: further selecting at least one speech synthesis model from a local first speech synthesis model and a server-side second speech synthesis model based on the network communication capability between the client and the server.

[0087] In some embodiments, selecting at least one speech synthesis model from a local first speech synthesis model and a server-side second speech synthesis model includes: selecting a local first speech synthesis model to perform speech synthesis on the response text in response to a network communication capability between the client and the server being lower than a first capability level; and selecting at least one speech synthesis model from a local first speech synthesis model and a server-side second speech synthesis model based on the speech conversion requirements of the response text in response to a network communication capability between the client and the server exceeding a second capability level.

[0088] In some embodiments, obtaining the response speech corresponding to the response text includes: in response to a first local speech synthesis model being selected, requesting the first speech synthesis model to perform speech synthesis on the response text to obtain the response speech; and in response to a second speech synthesis model being selected at the server, sending the response text to the server to request the second speech synthesis model to perform speech synthesis on the response text to obtain the response speech.

[0089] In some embodiments, method 400 further includes: in response to the selection of a first speech synthesis model and a second speech synthesis model, playing at least a portion of the first response speech after obtaining the first response speech generated using the first speech synthesis model; and in response to receiving the second response speech generated using the second speech synthesis model from the server, stopping the playback of the first response speech, determining a playback start point in the second response speech, the playback start point corresponding to the playback stop point of the first response speech, and starting the playback of the second response speech from the playback start point.

[0090] In some embodiments, the speech conversion requirements indicate at least one of the following: intonation requirements for the response text, timbre requirements for the response text, and speech quality requirements for the response text.

[0091] In some embodiments, obtaining the response text for a question voice of a target user includes: selecting at least one speech recognition model from a local first speech recognition model and a server-side second speech recognition model based on the network communication capabilities between the client and the server, for performing speech recognition on the question voice; determining the question text of the question voice, wherein the question text is generated after performing speech recognition on the question voice using the selected at least one speech recognition model; and determining the response text based on the question text.

[0092] In some embodiments, selecting at least one speech recognition model from a local first speech recognition model and a server-side second speech recognition model includes: selecting a local first speech recognition model to perform speech recognition on problematic speech in response to a network communication capability between the client and the server being lower than a third capability level; and selecting a first speech recognition model and / or a second speech recognition model to perform speech recognition on problematic speech in response to a network communication capability between the client and the server exceeding a fourth capability level.

[0093] In some embodiments, determining the problem text of the problematic speech includes: in response to the selection of a first speech recognition model and a second speech recognition model, requesting the first speech recognition model to perform speech recognition on the problematic speech, and sending the problematic speech to the server to request the second speech recognition model to perform speech recognition on the problematic speech; in response to obtaining a first candidate problem text generated by the first speech recognition model, determining the quality score of the first candidate problem text; and in response to the quality score of the first candidate problem text meeting the quality requirements, determining the first candidate problem text as the problem text of the problematic speech, and sending a stop generation instruction to the server to stop the speech synthesis of the problematic speech.

[0094] In some embodiments, obtaining the response text for a question spoken by a target user includes: determining the question text corresponding to the question spoken; selecting either an offline first question-and-answer model or a server-side second question-and-answer model based on the question complexity corresponding to the question text; and obtaining the response text corresponding to the question text, wherein the response text is generated based on the question text using the selected question-and-answer model.

[0095] Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. Figure 5 shows an exemplary structural block diagram of an apparatus 500 for voice processing according to some embodiments of this disclosure. The apparatus 500 may be implemented as or included in the client 110. The various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0096] As shown in Figure 5, the device 500 includes a response text acquisition module 510, configured to acquire the response text of a question voice for a target user. The device 500 also includes a synthesis model selection module 520, configured to select at least one speech synthesis model from a local first speech synthesis model and a server-side second speech synthesis model, based at least on the speech conversion requirements of the response text, for performing speech synthesis on the response text. The device 500 also includes a response voice acquisition module 530, configured to acquire the response voice corresponding to the response text, wherein the response voice is generated after performing speech synthesis on the response text using the selected at least one speech synthesis model.

[0097] In some embodiments, the synthesis model selection module 520 is further configured to: select at least one of a local first speech synthesis model and a server-side second speech synthesis model based on the network communication capability between the client and the server.

[0098] In some embodiments, the synthesis model selection module 520 is further configured to: in response to the network communication capability between the client and the server being lower than a first capability level, select a local first speech synthesis model for performing speech synthesis on the response text; and in response to the network communication capability between the client and the server exceeding a second capability level, select at least one of the local first speech synthesis model and the second speech synthesis model at the server based on the speech conversion requirements of the response text.

[0099] In some embodiments, the response speech acquisition module 530 is further configured to: in response to the selection of a local first speech synthesis model, request the first speech synthesis model to perform speech synthesis function on the response text to obtain the response speech; and in response to the selection of a second speech synthesis model at the server, send the response text to the server to request the second speech synthesis model to perform speech synthesis function on the response text to obtain the response speech.

[0100] In some embodiments, the apparatus 500 further includes: a first voice playback module configured to, in response to the selection of a first voice synthesis model and a second voice synthesis model, play at least a portion of the first response voice after obtaining a first response voice generated using the first voice synthesis model; and a second voice playback module configured to, in response to receiving a second response voice generated using the second voice synthesis model from a server, stop playing the first response voice, determine a playback start point in the second response voice, the playback start point corresponding to the playback stop point of the first response voice, and start playing the second response voice from the playback start point.

[0101] In some embodiments, the speech conversion requirements indicate at least one of the following: intonation requirements for the response text, timbre requirements for the response text, and speech quality requirements for the response text.

[0102] In some embodiments, the response text acquisition module 510 is further configured to: select at least one of a local first speech recognition model and a server-side second speech recognition model based on the network communication capability between the client and the server, for performing speech recognition on the question speech; determine the question text of the question speech, wherein the question text is generated after performing speech recognition on the question speech using the selected at least one speech recognition model; and determine the response text based on the question text.

[0103] In some embodiments, the response text acquisition module 510 is further configured to: select a local first speech recognition model to perform speech recognition on the question speech in response to the network communication capability between the client and the server being lower than the third capability level; and select a first speech recognition model and / or a second speech recognition model to perform speech recognition on the question speech in response to the network communication capability between the client and the server being higher than the fourth capability level.

[0104] In some embodiments, the response text acquisition module 510 is further configured to: in response to the selection of a first speech recognition model and a second speech recognition model, request the first speech recognition model to perform speech recognition function on the question speech, and send the question speech to the server to request the second speech recognition model to perform speech recognition function on the question speech; in response to acquiring the first candidate question text generated by the first speech recognition model, determine the quality score of the first candidate question text; and in response to the quality score of the first candidate question text meeting the quality requirements, determine the first candidate question text as the question text of the question speech, and send a stop generation instruction to the server to stop the speech synthesis of the question speech.

[0105] In some embodiments, the response text acquisition module 510 is further configured to: determine the question text corresponding to the question voice; select an offline first question-and-answer model or a server-side second question-and-answer model based on the question complexity corresponding to the question text; and acquire the response text corresponding to the question text, wherein the response text is generated based on the question text using the selected question-and-answer model.

[0106] The modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more modules may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 500 may be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0107] It should be understood that one or more steps in the above methods can be performed by appropriate electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may, for example, include the client 110 in Figure 1.

[0108] Figure 6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in Figure 6 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 600 shown in Figure 6 can be used to implement the client 110 of Figure 1 or the device 500 of Figure 5.

[0109] As shown in Figure 6, the electronic device 600 is in the form of a general-purpose electronic device. Components of the electronic device 600 may include, but are not limited to, one or more processing units or processors 610, memory 620, storage devices 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processor 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 620. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0110] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.

[0111] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 6, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0112] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0113] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0114] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0115] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0116] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0117] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some, as newer, implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0119] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A voice processing method applied to a client, the method comprising: Obtain the text of the voice response to the question posed to the target user; Based at least on the speech conversion requirements of the response text, at least one speech synthesis model is selected from the local first speech synthesis model and the server-side second speech synthesis model to perform speech synthesis function on the response text; as well as Obtain the response speech corresponding to the response text, wherein the response speech is generated by performing speech synthesis on the response text using at least one selected speech synthesis model.

2. The method according to claim 1, wherein selecting at least one speech synthesis model from a local first speech synthesis model and a server-side second speech synthesis model comprises: Based on the network communication capability between the client and the server, at least one speech synthesis model is selected from the first speech synthesis model locally and the second speech synthesis model at the server.

3. The method according to claim 2, wherein selecting at least one speech synthesis model from the local first speech synthesis model and the second speech synthesis model at the server comprises: In response to the fact that the network communication capability between the client and the server is lower than the first capability level, the local first speech synthesis model is selected to perform speech synthesis function on the response text; as well as In response to the network communication capability between the client and the server exceeding the second capability level, based on the speech conversion requirements of the response text, at least one speech synthesis model is selected from the first speech synthesis model locally and the second speech synthesis model at the server.

4. The method according to claim 1, wherein obtaining the response voice corresponding to the response text includes: In response to the selection of the first local speech synthesis model, the first speech synthesis model is requested to perform speech synthesis on the response text to obtain the response speech; as well as In response to the selection of the second speech synthesis model at the server, the response text is sent to the server to request the second speech synthesis model to perform speech synthesis on the response text to obtain the response speech.

5. The method according to claim 1, further comprising: In response to the selection of the first speech synthesis model and the second speech synthesis model, at least a portion of the first response speech is played after obtaining the first response speech generated using the first speech synthesis model; as well as In response to receiving a second response voice generated using the second speech synthesis model from the server, Stop playing the first response voice message. Determine the start point of playback in the second response voice, the start point of playback corresponding to the stop point of playback in the first response voice, and The second response voice will be played starting from the playback start point.

6. The method of claim 1, wherein the speech conversion requirement indicates at least one of the following: intonation requirements for the response text, timbre requirements for the response text, and speech quality requirements for the response text.

7. The method of claim 1, wherein obtaining the response text of the question spoken by the target user comprises: Based on the network communication capability between the client and the server, at least one speech recognition model is selected from the local first speech recognition model and the second speech recognition model at the server, and used to perform speech recognition function on the question speech. Determine the question text of the question speech, wherein the question text is generated after performing a speech recognition function on the question speech using at least one selected speech recognition model; as well as The response text is determined based on the question text.

8. The method of claim 7, wherein selecting at least one speech recognition model from a local first speech recognition model and a second speech recognition model at the server comprises: In response to the fact that the network communication capability between the client and the server is lower than the third capability level, a local first speech recognition model is selected to perform speech recognition function on the problematic speech; as well as In response to the network communication capability between the client and the server exceeding the fourth capability level, the first speech recognition model and / or the second speech recognition model are selected to perform speech recognition function on the problematic speech.

9. The method of claim 7, wherein determining the question text of the questioned speech comprises: In response to the selection of the first speech recognition model and the second speech recognition model, The request is made to the first speech recognition model to perform speech recognition on the question speech, and Send the question voice to the server to request that the second speech recognition model be used to perform speech recognition on the question voice; In response to obtaining the first candidate question text generated by the first speech recognition model, a quality score of the first candidate question text is determined. as well as The quality score of the first candidate question text meets the quality requirements. The first candidate question text is determined as the question text of the question speech, and Send a stop generation instruction to the server to stop the speech synthesis of the problematic speech.

10. The method of claim 1, wherein obtaining the response text of the question spoken by the target user comprises: Determine the text of the question corresponding to the spoken question; Based on the complexity of the question text, either the offline first question-answering model or the server-side second question-answering model is selected. as well as Obtain the response text corresponding to the question text, wherein the response text is generated based on the question text using the selected question-answering model.

11. An apparatus for voice processing, applied to a client, the apparatus comprising: The response text acquisition module is configured to acquire the response text for the target user's voice question. The synthesis model selection module is configured to select at least one of a local first speech synthesis model and a server-side second speech synthesis model, based at least on the speech conversion requirements of the response text, for performing speech synthesis functions on the response text. as well as The response speech acquisition module is configured to acquire the response speech corresponding to the response text, wherein the response speech is generated by performing speech synthesis on the response text using at least one selected speech synthesis model.

12. An electronic device, comprising: At least one processor; as well as At least one memory, coupled to the at least one processor and storing instructions for execution by the at least one processor, which, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 10.

14. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 10.

Images