Voice processing method and apparatus, and device, storage medium and program product

Abstract

Provided in the embodiments of the present disclosure are a voice processing method and apparatus, and a device, a storage medium and a program product. The method comprises: on the basis of voice collected during a voice recording process of an object, determining attribute information related to the object, wherein the attribute information at least indicates the speaking rate of the object; on the basis of the attribute information, determining configuration information of a pause duration threshold with respect to the object during the voice recording process; and on the basis of the configuration information of the pause duration threshold, controlling voice activity detection executed on the voice recording process. Therefore, the speaking rate of an object can be determined on the basis of collected voice, and a pause duration threshold when voice activity detection is executed can be autonomously adjusted on the basis of the speaking rate of the object. Different pause duration thresholds can be determined with respect to different objects, thereby facilitating an improvement in the individuality and adaptability of voice activity detection executed on different objects, and improving the accuracy of the pause duration thresholds.

Description

Methods, apparatus, devices, storage media, and program products for speech processing

[0001] This application claims priority to Chinese Patent Application No. 202410869821.2, filed on June 28, 2024, entitled "Method, Apparatus, Device, Storage Medium and Program Product for Speech Processing", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for speech processing. Background Technology

[0003] With the development of internet technology, more and more applications and platforms offer voice services, bringing numerous conveniences to users. Handheld or portable devices support voice interaction, such as mobile phones, tablets, and laptops; even some wearable devices support voice interaction. Voice activity detection (VAD) is a technology used in speech processing to detect the presence of voice signals. Applications and platforms with voice services can use VAD technology to detect the start and end of a user's speech, preventing noise or other sounds from being recorded and producing unexpected results. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for speech processing is provided. The method includes: determining object-related attribute information based on speech acquired during speech recording of an object, the attribute information indicating at least the object's speech rate; determining configuration information for a pause duration threshold for the object during speech recording based on the attribute information; and controlling speech activity detection during the speech recording process based on the pause duration threshold configuration information.

[0005] In a second aspect of this disclosure, an apparatus for speech processing is provided. The apparatus includes: an attribute information determination module configured to determine attribute information related to the object based on speech acquired during speech recording of the object, the attribute information indicating at least the speech rate of the object; a configuration information determination module configured to determine configuration information for a pause duration threshold for the object during speech recording based on the attribute information; and a speech detection control module configured to control speech activity detection during the speech recording process based on the pause duration threshold configuration information.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores computer-executable instructions, which, when executed by a processor, implement the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The product includes computer-executable instructions, wherein when executed by a processor, the computer-executable instructions implement the method according to a first aspect of this disclosure.

[0009] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1A shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 1B shows an example of speech activity detection;

[0013] Figure 2 illustrates a schematic diagram of an example architecture for speech processing according to some embodiments of the present disclosure;

[0014] Figure 3 shows an example of a fitting curve for the difference scaling factor according to some embodiments of the present disclosure;

[0015] Figure 4 shows a schematic diagram of a signaling flow for voice processing according to some embodiments of the present disclosure;

[0016] Figure 5 shows a flowchart of a method for speech processing according to some embodiments of the present disclosure;

[0017] Figure 6 illustrates an exemplary structural block diagram of a speech processing apparatus according to some embodiments of the present disclosure; and

[0018] Figure 7 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0021] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and authorization should be obtained from the relevant users. Among them, relevant users may include any type of rights holder, such as individuals, enterprises, and groups.

[0024] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly inform the user that the requested operation will require obtaining and using the user's information, thereby enabling the relevant user to choose whether to provide information to the software or hardware such as the electronic device, application, server, or storage medium that performs the operation of the technical solution disclosed herein based on the prompt message.

[0025] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide information to the electronic device.

[0026] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0027] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0028] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0029] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0030] Figure 1A illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Example environment 100A relates to a client device 110 and a server device 130. The client device 110 may, for example, have an application with natural language processing capabilities installed. The application can be any suitable type of application, such as a social application, a chat application, or a media application. The application at least supports voice-based services. The server device 130 can provide background services for this application with natural language processing capabilities.

[0031] Client device 110 and / or applications within client device 110 may include an audio acquisition unit 112 and a voice activity detection system 114. The audio acquisition unit 112 is configured to acquire the voice 125 of user 120 during voice recording. The audio acquisition unit 112 may be, for example, a microphone built into or external to client device 110. The voice activity detection system 114 is configured to control voice activity detection (VAD) during voice recording based on pause duration to determine the start and end of user speech. For example, the voice activity detection system 114 may determine whether the user's speech has ended based on the pause duration during voice recording. The determination of end is used to instruct the audio acquisition unit 112 whether to acquire the voice 125 of user 120.

[0032] Referring to Figure 1B, which illustrates an example 100B of speech activity detection. As shown in Figure 1B, the start and end of speech recording are generally referred to as VAD_START and VAD_END. Client device 110 can determine VAD_START and begin recording speech upon detecting speech 125 from user 120. Specifically, after the sound is recorded, spectral analysis can detect that human speech characteristics match predetermined feature targets. At this time, speech activity detection system 114 determines that VAD_START has been detected and instructs audio acquisition unit 112 to start recording. Client device 110 can begin VAD_END pause detection upon not detecting speech 125. Specifically, when the user stops speaking, speech activity detection system 114 waits for a period of time to monitor whether the user continues speaking. If no user speech is detected, speech activity detection system 114 issues a VAD_END signal. The duration of VAD_END pause detection can be referred to as pause duration. If the pause duration reaches the pause duration threshold, the voice activity detection system 114 can determine that the voice has ended, thereby stopping the voice recording and transferring the recorded voice for subsequent processing, etc.

[0033] In some embodiments, the client device 110 in FIG1A communicates with the server device 130 to provide voice processing services. The client device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the client 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0034] Server-side device 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server-side device 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0035] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0036] As can be seen from the above process of voice activity detection, the detection of the end of speech (VAD_END) during voice recording plays a crucial role in whether the voice system can respond to subsequent processes in a timely manner. For example, in many voice interaction scenarios, it is necessary to perform Automatic Speech Recognition (ASR) on the speech, and then determine the content of the response to the user and perform text-to-speech (TTS) based on the speech or ASR results. If the user cannot stop speaking in time, it will affect the response speed of subsequent actions, resulting in a long waiting time for the user.

[0037] Traditionally, speech activity detection systems often rely on fixed pause duration thresholds to control speech activity during recording. However, since different users and different types of users may pause for varying durations during recording, using a fixed threshold to determine when to stop recording will not be suitable for all users. For example, for users with slower speaking speeds, a low threshold might cause the system to stop recording before the user has finished speaking. Therefore, it is desirable to be able to set different pause duration thresholds for different types of users.

[0038] In view of this, according to embodiments of the present disclosure, an improved speech processing scheme is provided. According to the scheme of the embodiments of the present disclosure, based on the speech acquired during the speech recording process of an object, attribute information related to the object is determined, and the attribute information at least indicates the speech rate of the object. Based on the attribute information, configuration information for a pause duration threshold for the object during the speech recording process is determined. The speech activity detection during the speech recording process is controlled based on the configuration information of the pause duration threshold.

[0039] In this way, the speech rate of an individual can be determined based on the acquired speech, and the pause duration threshold for speech activity detection can be adjusted based on the speech rate. This allows for dynamic adjustment of the pause duration threshold required for speech activity detection for different individuals in different scenarios, improving the personalization and adaptability of speech activity detection for different individuals and enhancing the accuracy of the pause duration threshold.

[0040] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0041] Figure 2 illustrates a schematic diagram of an example architecture 200 for voice processing according to some embodiments of the present disclosure. The example architecture 200 relates to a client device 110 and a server device 130. For ease of discussion, the architecture 200 will be described with reference to the environment 100 of Figure 1. It should be noted that the operations performed by the aforementioned client device 110, and the operations performed by the client device 110 as described below, may specifically be performed by relevant applications installed on the client device 110. In some embodiments, operations performed on the electronic device may be completed with the assistance of the server device 130.

[0042] Client device 110 may include an audio acquisition unit 112, a speech activity detection system 114 (also referred to as a VAD system 114), a VAD adjustment system 212, and a speech processing system 214. For example, in response to receiving a speech recording request, client device 110 may instruct the audio acquisition unit 112 to acquire speech 125 from object 201. Object 201 may include, for example, a user (e.g., user 120), or may include a voice-producing object other than a human. Audio acquisition unit 112 may include, for example, a microphone.

[0043] Example architecture 200 also involves a VAD configuration system 230. The VAD configuration system 230 can be deployed on server device 130 or client device 110. If the VAD configuration system 230 is configured on client device 110, client device 110 can directly send the voice 125 captured during the voice recording of object 201 to the VAD configuration system 230. If the VAD configuration system 230 is configured on server device 130, client device 110 can, for example, send the voice 125 captured during the voice recording of object 201 to the voice service service 222 of server device 130, so that the voice service service 222 can send the voice 125 to the VAD configuration system 230.

[0044] The VAD configuration system 230 can determine attribute information related to object 201 based on voice 125. In some embodiments, the VAD configuration system 230 can determine attribute information based on voice 125 only if authorization is obtained. The VAD configuration system 230 can, for example, provide an authorization request interface to the user during the first interaction with the digital assistant or each time the digital assistant is activated, and receive the user's permission configuration via this authorization request interface. The authorization request interface may include, for example, permission prompt information and at least one operation control. The at least one operation control may include an authorization confirmation control. The permission prompt information may include, for example, the text "Allow the system to determine relevant information based on user voice?" The VAD configuration system 230 can determine that user authorization has been obtained in response to detecting a trigger on the confirmation control.

[0045] As described below, this attribute information is used to determine or adjust the pause duration threshold for speech activity detection of object 201. In some embodiments, the VAD configuration system 230 may utilize a machine learning model to determine this attribute information based on speech 125. This machine learning model may be based on any suitable model architecture, including but not limited to Transformer models, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep neural networks (DNNs), and so on. In some embodiments, the machine learning model may be based on a language model (LM). Language models, by learning from a large corpus, are capable of question-answering.

[0046] For example, as shown in FIG2, the VAD configuration system 230 may include a language model 232. The VAD configuration system 230 may provide speech 125 to the language model 232. The VAD configuration system 230 may, for example, determine a prompt for the language model 232 based on the speech 125. The prompt includes guidance information for the language model 232, which may, for example, be used to guide the model on how to use the speech 125 to generate attribute information. In some embodiments, speech processing services may be provided based on at least one trained machine learning model. At least one machine learning model may include one or more language models. The language model used for processing speech (e.g., a language model for performing speech synthesis TTS, a language model for performing speech recognition ASR, etc.) may be referred to as the main language model or language model, and the language model 232 used to determine attribute information may be referred to as the secondary language model.

[0047] The model output of language model 232 can, for example, indicate attribute information of object 201. This attribute information can at least indicate the speech rate of object 201. This is because the speech rate of an object affects the duration of pauses during speech, thereby affecting the setting of the pause duration threshold. Of course, the attribute information can include any appropriate information, and it can also indicate any appropriate information other than speech rate (e.g., the type of the object), as long as this information can affect the pause duration during the object's speech. In some embodiments, the attribute information can also indicate the speaking context associated with object 201. In different speaking contexts, the object may exhibit different speaking rates or pause durations. For example, in a speech or meeting context, the normal pause duration of the object's speech may be longer, in which case the pause duration threshold should be set larger. In a noisy context, the object's speech will be more rapid, in which case the pause duration threshold can be set smaller.

[0048] The VAD configuration system 230 may also include a VAD service module 234. The VAD service module 234 may, for example, determine configuration information for a pause duration threshold for the object 201 during voice recording based on determined attribute information. It should be noted that in some embodiments, the VAD configuration system 230 may also directly provide the attribute information to the client device 110, so that the client device 110 can determine the pause duration threshold configuration information based on the received attribute information.

[0049] VAD service module 234 can determine the configuration information for the pause duration threshold in any appropriate manner. For example, VAD service module 234 can use predetermined rules or algorithms to determine the configuration information for the pause duration threshold based on attribute information. Alternatively, VAD service module 234 can also use a trained machine learning model to determine the configuration information for the pause duration threshold based on attribute information related to object 201. This machine learning model and the machine learning model used to determine the attribute information can be the same model or different models. VAD service module 234 can, for example, provide the attribute information to the machine learning model and determine the configuration information for the pause duration threshold based on the model output of this machine learning model.

[0050] The VAD configuration system 230 can directly determine the pause duration threshold based on attribute information. In some embodiments, the configuration information for the pause duration threshold can directly indicate the specific value of the pause duration threshold. In this case, the VAD configuration system 230 can directly determine the pause duration threshold based on the configuration information. Alternatively or additionally, in some embodiments, the configuration information may also indicate adjustment information for the predetermined pause duration threshold, such as a threshold adjustment factor. The VAD configuration system 230 can, for example, adjust the predetermined pause duration threshold based at least on the threshold adjustment factor. The threshold adjustment factor may, for example, include a difference scaling factor (factor) for scaling, an increment factor (delta) for increasing or decreasing, and so on.

[0051] In some embodiments, taking the threshold adjustment factor as an example of the difference scaling factor, the VAD service module 234 can determine the difference scaling factor based on the object's attribute information and can adjust the predetermined pause duration threshold based on the difference scaling factor. In some embodiments, the difference scaling factor is used to scale the predetermined pause duration threshold to obtain the adjusted pause duration threshold. The fitting curve of the difference scaling factor can be configured as shown in Figure 3. Figure 3 shows an example 300 of the fitting curve of the difference scaling factor according to some embodiments of the present disclosure. The difference scaling factor can, for example, be in the range of (0, 1). The faster the speech rate indicated by the attribute information, the closer the value of the difference scaling factor is to 1, which means that the scaling degree of the predetermined pause duration threshold is smaller. The slower the speech rate indicated by the attribute information, the closer the value of the difference scaling factor is to 0, which means that the scaling degree of the predetermined pause duration threshold is larger.

[0052] In some embodiments, the VAD service module 234 can also determine the longest waiting time (T) during the voice recording process based on attribute information. f ) and minimum waiting time (T) sThe waiting time can be, for example, the interval between acquiring two adjacent speech units. The VAD service module 234 can, for example, determine the predetermined pause duration threshold using the following formula based on the difference scaling factor, the longest waiting time, and the shortest waiting time: t = (T f -T s )*factor (1) V t =T s +t (2)

[0053] VAD service module 234, for example, can use a threshold adjustment factor (factor) to determine the adjustment amount of the predetermined pause duration threshold based on formula (1); and adjust the predetermined pause duration threshold T based on formula (2). s It is understood that the above only provides an example of how to adjust the predetermined pause duration threshold. In other embodiments, the predetermined pause duration threshold can be adjusted in other ways.

[0054] In some embodiments, as mentioned above, the VAD service module 234 can directly determine the pause duration threshold to be used without needing to determine adjustment information for the predetermined pause duration threshold. In this case, the dynamically determined pause duration threshold can be directly used for speech activity detection.

[0055] In some embodiments, after determining the pause duration threshold for a specific object, the VAD configuration system 230 can provide the determined pause duration threshold to the client device 110. It should be noted that in some embodiments, the VAD configuration system 230 can also directly provide the pause duration threshold configuration information to the client device 110, so that the client device 110 can determine the pause duration threshold itself based on the pause duration threshold configuration information. For example, the client device 110 includes a VAD adjustment system 212.

[0056] The VAD adjustment system 212 can determine the pause duration threshold during speech detection based on configuration information of the pause duration threshold. In some embodiments, if the newly determined pause duration threshold differs from a previous pause duration threshold (e.g., a historical pause duration threshold determined based on historical speech), the VAD adjustment system 212 can also update the pause duration threshold, and the updated pause duration threshold is the newly determined pause duration threshold. The VAD adjustment system 212 can, for example, update the pause duration threshold in real time. The VAD adjustment system 212 can also, for example, determine the pause duration threshold to be adjusted in response to the satisfaction of a predetermined condition. The predetermined condition may include, for example, that the difference between the newly determined pause duration threshold and the previously used pause duration threshold is greater than a predetermined difference. The predetermined condition may also include, for example, that the time elapsed since the last update of the pause duration threshold until the time elapsed since the determination of the new pause duration threshold reaches a predetermined duration.

[0057] Client device 110 is configured to perform a voice recording process. Client device 110 can perform voice activity detection during the voice recording process based on a comparison between the pause duration detected during the voice recording process and a pause duration threshold. For example, the voice activity detection system 114 of client device 110 can detect pause durations during the voice recording process where no voice is detected, and stop acquiring voice when the pause duration reaches the pause duration threshold. The voice activity detection system 114 can, for example, provide the acquired voice 125 to the voice processing system 214. The voice processing system 214 can, for example, provide the acquired voice 125 to the voice service service 222 of server device 130. In some embodiments, the voice processing system 214 can also determine the type of voice processing service (e.g., speech recognition, speech synthesis, etc.) that object 201 wants to perform on the voice 125, and inform the voice service service 222 of this type.

[0058] Taking the determination of whether to perform a speech recognition service on speech 125 as an example, the voice service 222 may provide speech 125 to the speech recognition model 224, and may use the speech recognition model 224 to determine the text corresponding to speech 125. The voice service 222 may provide the text corresponding to speech 125 to the client device 110. The client device 110 may present a specific page (e.g., a session page) and provide the text corresponding to speech 125 to the object 201 based on the page.

[0059] In some embodiments, example architecture 200 may further include a VAD feedback system 242. Similar to VAD configuration system 230, VAD feedback system 242 can also be configured at server device 130 or client device 110. VAD feedback system 242 may, for example, acquire feedback information during the process of performing speech activity detection using a pause duration threshold. The feedback information may, for example, indicate to object 201 its level of satisfaction (e.g., satisfied or dissatisfied) with the performance of speech activity detection using the pause duration threshold, suggestions for adjustments to the performance of speech activity detection using the pause duration threshold (e.g., a desire to decrease or increase the pause duration threshold), and so on.

[0060] The VAD feedback system 242 can, for example, adjust the determination of object-related attribute information based on feedback information. Exemplarily, the VAD feedback system 242 can instruct the VAD configuration system 230 to adjust the method of determining attribute information, adjust the parameters of the language model 232, etc., to determine more accurate attribute information. The VAD feedback system 242 can also, for example, adjust the determination of pause duration threshold configuration information based on feedback information. Exemplarily, the VAD feedback system 242 can instruct the VAD configuration system 230 to adjust the method of determining the pause duration threshold configuration information, adjust the method of determining the threshold adjustment factor, etc., to determine more accurate pause duration threshold configuration information, thereby adaptively determining the pause duration threshold to be used.

[0061] Figure 4 illustrates a schematic diagram of a signaling stream 400 for voice processing according to some embodiments of the present disclosure. In the signaling stream 400, a voice activity detection system 114 may initiate a voice recording process in response to receiving (401) a voice recording request from an object 201. During the voice recording process, the voice activity detection system 114 may, for example, provide (402) the voice collected during the voice recording process to the server device 130 in response to a pause duration reaching a predetermined pause duration threshold.

[0062] Server device 130 can provide (403) speech to language model 232 so that language model 232 can determine attribute information related to object 201 based on the speech collected during the speech recording of object 201, the attribute information indicating at least the speech rate of object 201. Server device 130 can also perform corresponding speech processing on the speech based on the type of speech processing service to be performed on the speech of object 201. For example, if it is determined that ASR / TTS should be performed on the speech of object 201, server device 130 can perform ASR / TTS on the speech using ASR model / TTS model (404). Server device 130 can provide (405) the text obtained from performing ASR / TTS on the speech to speech activity detection system 114. Speech activity detection system 114 can be configured at client device 110, for example, and client device 110 can provide the text via a specific page.

[0063] Language model 232 can, for example, provide (406) determined attribute information to speech activity detection system 114. Client device 110 can, for example, determine configuration information for pause duration thresholds for an object during speech recording based on the determined attribute information, and can determine the pause duration threshold based on the configuration information. If the newly determined pause duration threshold is different from the previous pause duration threshold, client device 110 can also adjust the pause duration threshold, and the adjusted pause duration threshold is also the newly determined pause duration threshold.

[0064] The speech activity detection system 114 can, for example, control the speech activity detection during the speech recording process based on an adjusted pause duration threshold. The speech activity detection system 114 can, for example, provide (408) newly acquired speech (i.e., adjusted speech) during the speech recording process to the server device 130 in response to a pause duration reaching the adjusted pause duration threshold. Similarly, the server device 130 can provide (409) the text obtained by performing speech processing on the speech to the speech activity detection system 114.

[0065] Client device 110 may, for example, receive feedback information during the process of performing speech activity detection using an adjusted pause duration threshold. Speech activity detection system 114 in client device 110 may, for example, provide the feedback information (410) to VAD feedback system 242. VAD feedback system 242 may, for example, adjust the determination of configuration information for object attribute information / pause duration threshold based on the feedback information. VAD feedback system 242 may, for example, determine a specific adjustment method / adjustment suggestion for the configuration information based on the feedback information, and may provide the adjustment method / adjustment suggestion (411) to speech activity detection system 114. Speech activity detection system 114 may, for example, adjust the determination of configuration information based on the received adjustment method / adjustment suggestion. VAD feedback system 242 may also, for example, determine a specific adjustment method / adjustment suggestion for attribute information based on the feedback information, and may provide the adjustment method / adjustment suggestion (412) to language model 232. Language model 232 may, for example, adjust the determination of attribute information based on the received adjustment method / adjustment suggestion.

[0066] In summary, according to the embodiments of this disclosure, the speech rate of an object can be determined based on the acquired speech, and the pause duration threshold during speech activity detection can be automatically adjusted based on the object's speech rate. Different pause duration thresholds can be determined for different objects, which helps to improve the personalization and adaptability of speech activity detection for different objects and improve the accuracy of the pause duration thresholds.

[0067] Figure 5 illustrates a flowchart of a method 500 for voice processing according to some embodiments of the present disclosure. Method 500 can be implemented at a client device 110 and / or a server device 130.

[0068] In box 510, client device 110 and / or server device 130 determine object-related attribute information based on the speech collected during the speech recording of the object, the attribute information indicating at least the speech rate of the object.

[0069] In box 520, client device 110 and / or server device 130 determine configuration information for a pause duration threshold for an object during voice recording based on attribute information.

[0070] In box 530, client device 110 and / or server device 130 control the detection of voice activity during the voice recording process based on configuration information of pause duration threshold.

[0071] In some embodiments, method 500 is implemented at client device 110, which is configured to perform a voice recording process, and wherein controlling voice activity detection of the voice recording process based on configuration information of a pause duration threshold includes: determining a pause duration threshold for a target based on the configuration information; and performing voice activity detection of the voice recording process based on a comparison between a pause duration detected during the voice recording process and the pause duration threshold.

[0072] In some embodiments, method 500 is implemented at server device 130, and wherein controlling voice activity detection of the voice recording process based on configuration information of a pause duration threshold includes: sending the configuration information of the pause duration threshold to a client device, so that the client device determines the pause duration threshold based on the configuration information, the client device being configured to perform the voice recording process.

[0073] In some embodiments, determining attribute information related to an object includes: using a machine learning model to determine attribute information based on collected speech.

[0074] In some embodiments, the configuration information indicates a threshold adjustment factor, and wherein the pause duration threshold for an object is determined by adjusting a predetermined pause duration threshold based at least on the threshold adjustment factor.

[0075] In some embodiments, method 500 further includes: obtaining feedback information during the process of performing speech activity detection using a pause duration threshold; and adjusting at least one of the following based on the feedback information: determining object-related attribute information or determining configuration information for the pause duration threshold.

[0076] Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. FIG6 shows an exemplary structural block diagram of an apparatus 600 for voice processing according to some embodiments of this disclosure. The apparatus 600 may be implemented as or included in client device 110 and / or server device 130. The various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0077] As shown in Figure 6, the device 600 includes an attribute information determination module 610, configured to determine attribute information related to the object based on the speech collected during the object's speech recording process. The attribute information at least indicates the object's speech rate. The device 600 also includes a configuration information determination module 620, configured to determine configuration information for a pause duration threshold for the object during speech recording based on the attribute information. The device 600 further includes a speech detection control module 630, configured to control speech activity detection during the speech recording process based on the pause duration threshold configuration information.

[0078] In some embodiments, the apparatus 600 is implemented at a client device 110, which is configured to perform a voice recording process, and the voice detection control module 630 includes: a pause duration threshold determination module, configured to determine a pause duration threshold for a target based on configuration information; and a voice activity detection execution module, configured to perform voice activity detection of the voice recording process based on a comparison between the pause duration detected during the voice recording process and the pause duration threshold.

[0079] In some embodiments, the apparatus 600 is implemented at the server device 130, and the voice detection control module 630 includes: a configuration information sending module configured to send configuration information of a pause duration threshold to the client device, so that the client device can determine the pause duration threshold based on the configuration information, and the client device is configured to perform a voice recording process.

[0080] In some embodiments, the attribute information determination module 610 is further configured to: determine attribute information based on the collected speech using a machine learning model.

[0081] In some embodiments, the configuration information indicates a threshold adjustment factor, and wherein the pause duration threshold for an object is determined by adjusting a predetermined pause duration threshold based at least on the threshold adjustment factor.

[0082] In some embodiments, the apparatus 600 further includes: a feedback information acquisition module configured to acquire feedback information during the process of performing speech activity detection using a pause duration threshold; and an adjustment module configured to adjust at least one of the following based on the feedback information: the determination of object-related attribute information, or the determination of configuration information for the pause duration threshold.

[0083] The units and / or modules included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 600 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0084] It should be understood that one or more steps in the above methods can be performed by appropriate electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may, for example, include the client device 110 and / or the server device 130 in FIG1.

[0085] Figure 7 illustrates a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 700 shown in Figure 7 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 700 shown in Figure 7 can be used to implement the client device 110 and / or server device 130 of Figure 1.

[0086] As shown in Figure 7, the electronic device 700 is in the form of a general-purpose electronic device. Components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0087] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0088] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 7, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0089] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0090] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0091] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0092] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0093] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0094] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some newer implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0096] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for speech processing, comprising: Based on the speech collected during the speech recording of the object, the attribute information related to the object is determined, and the attribute information at least indicates the speech rate of the object; Based on the attribute information, configuration information for the pause duration threshold of the object during the voice recording process is determined; as well as The voice activity detection during the voice recording process is controlled based on the configuration information of the pause duration threshold.

2. The method of claim 1, wherein the method is implemented at a client device configured to perform the voice recording process, and wherein controlling the voice activity detection of the voice recording process based on configuration information of the pause duration threshold comprises: Based on the configuration information, determine the pause duration threshold for the object; as well as Based on a comparison between the pause duration detected during the voice recording process and the pause duration threshold, voice activity detection is performed on the voice recording process.

3. The method of claim 1, wherein the method is implemented at a server device, and wherein controlling the voice activity detection of the voice recording process based on configuration information of the pause duration threshold comprises: The configuration information of the pause duration threshold is sent to the client device, so that the client device can determine the pause duration threshold based on the configuration information, and the client device is configured to perform the voice recording process.

4. The method according to claim 1, wherein determining the attribute information related to the object includes: The attribute information is determined based on the collected speech using a machine learning model.

5. The method of claim 1, wherein the configuration information indicates a threshold adjustment factor, and wherein the pause duration threshold for the object is determined by adjusting a predetermined pause duration threshold based at least on the threshold adjustment factor.

6. The method according to claim 1, further comprising: Obtain feedback information during the process of performing the speech activity detection using the pause duration threshold; as well as Based on the feedback information, adjust at least one of the following: Determining the attribute information related to the object, or Determining the configuration information for the pause duration threshold.

7. An apparatus for speech processing, comprising: The attribute information determination module is configured to determine attribute information related to the object based on the speech collected during the speech recording process of the object, wherein the attribute information at least indicates the speech rate of the object; The configuration information determination module is configured to determine configuration information for the pause duration threshold of the object during the voice recording process based on the attribute information; as well as The voice detection control module is configured to control the voice activity detection during the voice recording process based on configuration information of the pause duration threshold.

8. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 6 when executed by the at least one processor.

9. A computer-readable storage medium having stored thereon computer-executable instructions, which can be executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 6.

Images