Voice processing method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a voice processing method and device, equipment, a storage medium and a product. The method comprises the following steps: in response to a first type of operation on a voice input area, starting a first input mode to receive voice input; in the first input mode, in response to a first operation of a second type on the voice input area, switching to a second input mode to receive voice input; and presenting a voice recognition result corresponding to the received voice input. Therefore, different voice input modes can be flexibly switched. Therefore, the user can input various contents for different purposes through voice, so that the convenience and practicability of voice input are improved, and better use experience is provided for the user.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, storage media, and products for speech processing. Background Art

[0002] Voice input is a common input method in various applications such as entertainment and social networking. The voice input by the user can be converted into text and other forms through the voice recognition model. However, the current voice input method is very simple. How to improve the richness and practicality of voice input is a technical problem that needs to be explored at present. Summary of the invention

[0003] In a first aspect of the present disclosure, a speech processing method is provided. The method comprises: in response to a first type of operation on a speech input area, starting a first input mode to receive speech input; in response to a second type of first operation on the speech input area in the first input mode, switching to a second input mode to receive speech input; and presenting a speech recognition result corresponding to the received speech input, in which a first recognition result corresponding to the speech received in the first input mode is distinguishable from a second recognition result corresponding to the speech received in the second mode.

[0004] In a second aspect of the present disclosure, a device for speech processing is provided. The device includes: a startup module configured to start a first input mode to receive speech input in response to a first type of operation on a speech input area; a first switching module configured to switch to a second input mode to receive speech input in response to a second type of first operation on the speech input area in the first input mode; and a result presentation module configured to present a speech recognition result corresponding to the received speech input, in which a first recognition result corresponding to the speech received in the first input mode is distinguishable from a second recognition result corresponding to the speech received in the second mode.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, wherein the computer executable instructions implement the method of the first aspect when executed by a processor.

[0008] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] FIG. 2A to FIG. 2D A schematic diagram showing a conversation interface with a virtual object according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram showing a process of changing the presentation style of a speech input area according to some embodiments of the present disclosure;

[0013] Figure 4 A schematic diagram of a process for generating speech recognition results according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic diagram showing the interaction process between an ASR manager, an ASR root session, a voice server, and an ASR subsession according to some embodiments of the present disclosure;

[0015] Figure 6 A flowchart showing a process for speech processing according to some embodiments of the present disclosure is shown;

[0016] Figure 7 A block diagram showing an apparatus for speech processing according to some embodiments of the present disclosure; and

[0017] Figure 8 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0019] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0020] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0022] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0025] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0026] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0027] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. In this article, "model" may also be referred to as "machine learning model", "machine learning network" or "network", which are used interchangeably in this article. A model may also include different types of processing units or networks.

[0028] As briefly mentioned above, voice input is a common input method in various applications such as entertainment and social networking. The voice input by the user can be converted into text and other forms through a voice recognition model. However, the current voice input method is very single and can only process the voice input by the user as a whole. When the voice input by the user contains multiple parts for different purposes, it is impossible to distinguish them. Taking the interaction between users and virtual objects as an example, users need to input the content of the conversation through voice, and sometimes they may also need to input non-conversation content such as actions, expressions, and emotions so that the virtual object can better understand the content of the conversation. However, the current voice input method cannot distinguish between the conversation content and non-conversation content in the voice. This has a certain impact on the convenience, practicality and user experience of voice input.

[0029] Embodiments of the present disclosure propose a scheme for speech processing. According to various embodiments of the present disclosure, in response to a first type of operation on a speech input area, a first input mode is started to receive speech input. Then, in the first input mode, in response to a second type of first operation on the speech input area, a switch is made to a second input mode to receive speech input. Finally, a speech recognition result corresponding to the received speech input is presented, in which a first recognition result corresponding to the speech received in the first input mode is distinguishable from a second recognition result corresponding to the speech received in the second mode.

[0030] In this way, different voice input modes can be flexibly switched. Voice recognition is performed on voices received in different input modes, and the presentation of recognition results in different input modes is distinguishable. This allows users to input a variety of content for different purposes through voice, thereby improving the convenience and practicality of voice input and providing users with a better user experience.

[0031] Example Environment

[0032] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the example environment 100, an application 120 is installed in a terminal device 110. A user 140 can interact with the application 120 via the terminal device 110 and / or an attached device of the terminal device 110.

[0033] In some embodiments, application 120 may be any appropriate application that can provide voice input related services. As an example, application 120 may be an instant messaging (IM) application. As another example, application 120 may be an entertainment application, such as a virtual object interaction application. It should be understood that the above are only examples of application 120 and are not intended to be limiting. Application 120 may be any appropriate type of application that supports voice input. Figure 1 In the environment 100, if the application 120 is active, the terminal device 110 can present the interface 150 of the application 120. The interface 150 may include various pages that the application 120 can provide, such as an interface for communicating with other users, a dialogue interface with a virtual object, a selection interface for a virtual object, and the like.

[0034] In some embodiments, the terminal device 110 communicates with the server 130 to provide the service of the application 120. The terminal device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface for the user (such as "wearable" circuits, etc.). The server 130 can be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0035] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0036] Some example embodiments of the present disclosure will continue to be described with reference to the accompanying drawings. It should be understood that the pages shown in the accompanying drawings are merely examples, and various page designs may actually exist. The various graphic elements in the page may have different arrangements and different visual representations, one or more of which may be omitted or replaced, and one or more other elements may also exist. The embodiments of the present disclosure are not limited in this respect. In addition, hereinafter, the example embodiments will be described primarily with respect to the terminal device 110. It should be understood that the actions described with respect to the terminal device 110 may be performed by the application 120 on the terminal device 110, or may be performed by the application 120 in collaboration with its service end (e.g., server 130).

[0037] First reference FIG. 2A to FIG. 2D Example embodiments of the present disclosure for speech processing are described. FIG. 2A to FIG. 2D Schematic diagrams of conversation interfaces 200A to 200D with virtual objects according to some embodiments of the present disclosure are shown.

[0038] It should be noted that the method for speech processing disclosed in the present invention can be applied to a variety of applications such as entertainment, IM, and social networking. The object targeted by the speech input can be any suitable type of object, such as other users or virtual objects. In this article, the virtual object can be, for example, a digital robot implemented with the help of a machine learning model (e.g., a language model), such as a digital assistant or a virtual chat object. For ease of understanding, the following mainly uses interactive applications with virtual objects as an example for explanation, but it should be understood that this is only exemplary and is not intended to be limiting. The embodiments for speech processing described with reference to interactive applications with virtual objects can be applied to other types of applications.

[0039] In some embodiments, the terminal device 110 may present a dialogue interface between the user 140 and the virtual object. Figure 2A Taking the example of a conversation with a virtual object A, an example of a conversation interface 200A with a virtual object is shown. The user 140 can have a conversation with the virtual object A in text, voice, etc. through the interface 200A.

[0040] In some embodiments, the interface 200A includes a voice input area 201. The user 140 can provide voice input by operating the voice input area 201 to communicate with the virtual object A. In some embodiments, if the first type of operation of the user 140 on the voice input area 201 is detected, the terminal device 110 can start the first input mode to receive the voice input. For example, the voice input area 201 may include a voice input control, and the first type of operation may be a pressing operation. The user 140 can start the first input mode by pressing the voice input control. In this input mode, the user can release the pressing of the voice input control to send the voice, or slide up to cancel the voice input. Taking the user's dialogue with the virtual object A as an example, the first input mode may be a dialogue content input mode. In this mode, the user can enter dialogue content such as "Hello".

[0041] In some embodiments, in the first input mode, the voice input area 201 is presented in a first style. Figure 2B An example of a dialogue interface 200B with a virtual object is shown. In the interface 200B, the voice input area 201 is presented in a first style. In this input mode, the voice input area 201 of the first style includes a black input box and two gray circles.

[0042] In some embodiments, in the first input mode, if the user 140 detects the first operation of the second type on the voice input area 201, the terminal device 110 can switch to the second input mode to receive the voice input. For example, the first operation of the second type can be a sliding operation. The user 140 can switch the first input mode to the second input mode by sliding to the left in the voice input area 201. Taking the user's dialogue with the virtual object A as an example, the second input mode can be an input mode for non-dialogue content such as actions, expressions, and emotions. In this mode, the user can input non-dialogue content such as "smiling" and "waving".

[0043] In some embodiments, in the second input mode, the voice input area 201 is presented in a second style. Figure 2C An example of a dialogue interface 200C with a virtual object is shown. In interface 200C, a voice input area 201 is presented in a second style. In this input mode, the second style voice input area 201 includes a white input box and two gray circles. Usually, the second style can be distinguished from the first style in terms of color, shape, etc., so that the user 140 can distinguish the current input mode.

[0044] In some embodiments, the terminal device 110 may present prompt information above the voice input area 201. Figure 2B and Figure 2CAs shown, prompt information 202 may be presented above the voice input area 201. Taking the user having a conversation with the virtual object A as an example, in the conversation content input mode, the prompt information 202 may be set to "Please enter a conversation", and in the non-conversation content input mode, the prompt information 202 may be set to "Please enter an action".

[0045] In some embodiments, in the second input mode, if a second operation of the second type of the user 140 on the voice input area 201 is detected, the terminal device 110 can switch to the first input mode to receive the voice input. The second operation is of the same type as the first operation, but in the opposite direction. For example, when the first operation is a sliding operation to the left, the second operation can be a sliding operation to the right. In some embodiments, the user 140 can also stop operating the voice input area in the second input mode to end the voice input and send the voice without returning to the first input mode.

[0046] In some embodiments, the terminal device 110 can present a speech recognition result corresponding to the received speech input. The speech recognition result can be presented after the speech input ends, or it can be presented gradually while the speech input is being input. In the speech recognition result, the first recognition result corresponding to the speech received in the first input mode is distinguishable from the second recognition result corresponding to the speech received in the second mode. The first recognition result and the second recognition result can include any suitable type of content, such as text, symbols, etc. The presentation effects of the recognition results in different input modes can be distinguished in any suitable manner.

[0047] In some embodiments, the recognition results in different input modes can be separated by delimiters. In such an embodiment, the recognition results in different input modes can have the same type of content (for example, all text), or can have different types of content (for example, the recognition result in one input mode is text, the recognition result in another input mode is a symbol, etc.). Exemplarily, the second recognition result can have a first delimiter and a second delimiter at the beginning and end, respectively. For example, the first delimiter can be a left bracket "(", and the second delimiter can be a right bracket ")". Figure 2D An example of a dialogue interface 200D with a virtual object is shown. In interface 200D, the voice recognition result "Hello (with a smile)" is presented in voice input area 201. "Hello" is the first recognition result corresponding to the voice received in the first input mode (conversation content input mode), and "with a smile" is the second recognition result corresponding to the voice received in the second input mode (non-conversation content input mode). The non-conversation content in the voice recognition result is distinguished from the conversation content by left and right brackets. Usually, the punctuation before and after the brackets is no longer presented.

[0048] Alternatively or additionally, in some embodiments, the recognition results in different input modes may be different types of content. Exemplarily, the first recognition result may include text corresponding to the voice received in the first input mode, and the second recognition result may include symbols (such as emoticons, action symbols, etc.) that match the voice received in the second input mode. For example, the text corresponding to the voice received in the second input mode can be first identified, and the symbol can be obtained by semantic understanding of the recognized text. Continuing with the above example, the "smiling" in the second recognition result can be presented with the expression "smiling" based on semantic understanding. The presented symbol (e.g., emoticon or action symbol) can be selected from a plurality of symbols obtained in advance, or can be generated based on semantic understanding. In this article, the embodiments are mainly described with text as the recognition result, but it should be understood that this is only exemplary and not intended to be limiting.

[0049] In some embodiments, the voice input area 201 is composed of Figure 2B The first style shown is transformed to Figure 2C In the process of the second style shown in FIG. 1 , the terminal device 110 may present the first preset animation effect. Figure 2C The second style shown transitions to Figure 2B During the process of the first style shown, the terminal device 110 can present a second preset animation. In some embodiments, the operation that triggers the switch between the first input mode and the second input mode can be a sliding operation. The presentation of the first preset animation or the second preset animation can be in response to the sliding operation satisfying a preset condition. The preset condition may include a sliding distance of the sliding operation being greater than or equal to a preset distance, for example, a sliding distance being greater than or equal to 50pt. Alternatively or additionally, the preset condition may include a sliding speed of the sliding operation being greater than or equal to a preset speed, for example, a sliding speed being greater than or equal to 100pt / s.

[0050] refer to Figure 3 An example of a process 300 for converting a speech input region style is shown. Figure 3As shown, in the dialogue content input mode, the voice input area is presented in the first style shown in 301. The circles on the left and right sides of the input box can be used to guide the user to switch the input mode. When the user 140 performs a side swipe (306) operation (i.e., the first operation mentioned above) on the voice input area, and the sliding distance of the side swipe operation reaches 50pt or the sliding speed reaches 100pt / s, the terminal device 110 is triggered (307) to switch to the non-dialogue content input mode. During the switching process, the left circle of the voice input area is left-aligned in the manner shown in 302 and gradually increases by 50 to 130pt (i.e., the first preset dynamic effect mentioned above), and the duration of the first preset dynamic effect can be 100ms. When switching to the non-dialogue input mode, the voice input area is presented in the second style shown in 303. When the user 140 performs a back swipe (308) operation (i.e., the second operation mentioned above) on the voice input area, and the sliding distance of the back swipe operation reaches 50pt or the sliding speed reaches 100pt / s, the terminal device 110 is triggered (309) to switch to the dialogue content input mode. During the switching process, the left circle of the voice input area is left-aligned in the manner shown in 304 and gradually shrinks to 30pt (i.e., the second preset animation effect mentioned above), and the duration of the second animation effect can be 100ms. When switching back to the non-dialogue input mode, the voice input area is presented in the first style shown in 305. During the presentation of the first and second animation effects, the size, color and other parameters of the left and right circles can be designed as needed. It should be noted that the above trigger mode switching operations and the animation effects during the switching process are only exemplary and are not intended to be limiting.

[0051] In some embodiments, if it is detected that the current input mode is switched from the first input mode to the second input mode, the terminal device 110 may provide the first voice received in the first input mode to the voice recognition model in the server 130 to obtain a first recognition result corresponding to the first voice. In some embodiments, if it is detected that the current input mode is switched from the second input mode to the first input mode, the terminal device 110 may provide the second voice received in the second input mode to the voice recognition model in the server to obtain a second recognition result corresponding to the second voice. In the case where both the first recognition result and the second recognition result are text, during the process of switching from the first input mode to the second input mode and from the second input mode to the first input mode, the terminal device 110 may respond to the sliding operation and send a corresponding instruction to the server 130 through the software development function block, where the marked text will generate a separator.

[0052] In some embodiments, the server 130 may combine the first recognition result and the second recognition result as at least a part of the speech recognition result. When the terminal device 110 receives multiple first speech segments or multiple second speech segments, the order in which the recognition results corresponding to the speech segments are combined is consistent with the input order of the corresponding speech.

[0053] refer to Figure 4 An example of a process 400 for generating speech recognition results is shown. Figure 4 As shown, in the process of a voice input, based on the switching of the input mode, the voice is divided into 5 sessions and processed separately, namely, session-1 in box 401 to session-5 in box 405. Session-1, session-3 and session-5 are sessions obtained in the first input mode, and session-2 and session-4 are sessions obtained in the second input mode. Multiple sessions can exist at the same time, for example, at the moment shown by the dotted line 407, session-2 to session-4 exist at the same time. Each session receives a segment of voice by the recorder, and then obtains a corresponding segment of text using the speech recognition model. The server 130 inserts a left bracket and a right bracket at the beginning and end of the "2-text" and "4-text" obtained in the second input mode, respectively, and merges the segments of text to obtain the speech recognition result ASR (Automatic Speech Recognition, speech recognition)-text, where ASR-text=1-text+(2-text)+3-text+(4-text)+5-text. It should be understood that the text in brackets can be presented as corresponding symbols.

[0054] In order to more clearly understand the speech processing scheme of the present disclosure, refer to Figure 5 An example of an interaction process 500 between an ASR manager, an ASR root session, a voice server, and an ASR subsession is described.

[0055] exist Figure 5 In the example, the ASR manager 520 may be a method or operation in the terminal device 110 , and the voice server 530 may be a method or operation in the server 130 .

[0056] Specifically, Figure 5 The interactive process 500 among the ASR manager 520, the ASR root session 525, the voice server 530, and the ASR subsession 535 during a speech input by the user 140 is shown. Figure 5As shown, the user presses (501) the input box through the ASR manager 520 to start the first input mode, such as the voice input mode. The ASR root session 525 creates (502) a voice recorder to collect sound, and initializes (503) a software development function block. The ASR root session 525 establishes (504) a connection to the voice server 530 and streams (505) the audio. After obtaining the text based on the audio recognition, the voice server 530 returns (506) the recognition result to the ASR root session 525.

[0057] continue Figure 5 In the process, when the user switches the input mode by sliding or other operations, such as inputting (507) action, the ASR root session 525 creates (508) an ASR subsession 535. It can be understood that each time the user switches the input mode, the ASR root session 525 will create an ASR subsession 535. The ASR subsession 535 also creates (509) a voice recorder to collect sound, and initializes (510) a software development function block. The ASR subsession 535 establishes (511) a connection to the voice server 530 and streams (512) the audio. After the voice server 530 obtains the text based on each audio segment recognition, it returns (513) the recognition result to the ASR subsession 535. The sub-ASR results of one or more ASR subsessions 535 are aggregated (514) to obtain a speech recognition result. The ASR root session 525 returns (515) the recognition result to the ASR manager 520, and the speech recognition result is presented in the terminal device 110. Finally, the ASR manager 520 sends (516) information on releasing resources to the ASR root session 525, and the ASR root session 525 sends (517) information on releasing resources to each ASR sub-session 535, thereby completing a speech input process.

[0058] In some embodiments, the voice input received by the terminal device 110 includes one or more first voice segments received in the first input mode and one or more second voice segments received in the second input mode, and the voice recognition result includes one or more first recognition results corresponding to one or more first voice segments and one or more second recognition results corresponding to one or more second voice segments. It should be noted that after starting the first input mode, the user may not input the first voice, and the first voice segment received by the terminal device 110 is the second voice. For example, in the process of a voice input, action content such as "(waving)" can be received at the beginning.

[0059] In some embodiments, the voice input is issued by the user to the virtual object during the interaction between the user and the virtual object, and one or more first recognition results include the user's dialogue content to the virtual object, and one or more second recognition results include explanation information of the dialogue content.

[0060] In some embodiments, the server 130 can generate a response to the voice input using a language model based on the voice recognition result. The second recognition result in the voice recognition result is used to assist the language model in understanding the first recognition result adjacent to (i.e., before or after) the second recognition result. Taking the interactive application with a virtual object as an example, the first recognition result can be the conversation content "I am very happy today" input by the user, and the second recognition result can be the expression "(frustrated)" input by the user. When replying using the voice recognition model, the expression "(frustrated)" can assist the model in understanding the real emotion implied by the user in the conversation "I am very happy today".

[0061] In some embodiments, the server 130 may generate a first part of the model prompt information based on one or more first recognition results, and the first part is used to provide the content of the conversation. The server 130 may also generate a second part of the model prompt information based on one or more second recognition results, and the second part is used to provide explanation information for the content of the conversation. The server 130 may provide the model prompt information to a language model to obtain an output of the language model, and determine a response based on the output of the language model. The language model may be trained based on sample conversation content of a user and a virtual object.

[0062] In summary, through the present disclosure, different voice input modes can be flexibly switched, and distinguishable voice recognition results can be presented according to the voice received in different input modes, so that users can input a variety of content for different purposes through voice, which improves the convenience and practicality of voice input and provides users with a better user experience.

[0063] Example Process

[0064] Figure 6 A flowchart of a process 600 for speech processing according to some embodiments of the present disclosure is shown. The process 600 may be implemented at the terminal device 110 or by the terminal device 110 coordinating the server 130. The process 600 is described below with reference to 1.

[0065] In block 610 , in response to a first type of operation on a voice input area, the terminal device 110 initiates a first input mode to receive a voice input.

[0066] At block 620 , in the first input mode, in response to a first operation of a second type on the voice input area, the terminal device 110 switches to a second input mode to receive a voice input.

[0067] In box 630, the terminal device 110 presents a speech recognition result corresponding to the received speech input, in which the first recognition result corresponding to the speech received in the first input mode is distinguishable from the second recognition result corresponding to the speech received in the second mode.

[0068] In some embodiments, the second recognition result has a first delimiter and a second delimiter at the beginning and the end, respectively.

[0069] In some embodiments, the first recognition result includes text corresponding to speech received in the first input mode, and the second recognition result includes symbols matching speech received in the second input mode.

[0070] In some embodiments, process 600 further includes: in the second input mode, in response to a second operation of a second type on the voice input area, switching to the first input mode to receive voice input, wherein the second operation is in an opposite direction to the first operation.

[0071] In some embodiments, in the first input mode, the voice input area is presented in a first style, and in the second input mode, the voice input area is presented in a second style.

[0072] In some embodiments, process 600 also includes at least one of the following: presenting a first preset animation effect during the transition of the voice input area from the first style to the second style, or presenting a second preset animation effect during the transition of the voice input area from the second style to the first style.

[0073] In some embodiments, the operation that triggers the switching between the first input mode and the second input mode is a sliding operation, and the presentation of the first preset animation effect or the second preset animation effect in response to the sliding operation satisfies at least one of the following: the sliding distance of the sliding operation is greater than or equal to the preset distance, or the sliding speed of the sliding operation is greater than or equal to the preset speed.

[0074] In some embodiments, the speech recognition result is generated in the following manner: in response to switching from a first input mode to a second input mode, providing a first speech received in the first input mode to a speech recognition model to obtain a first recognition result corresponding to the first speech; in response to switching from the second input mode to the first input mode, providing a second speech received in the second input mode to a speech recognition model to obtain a second recognition result corresponding to the second speech; and merging the first recognition result and the second recognition result as at least part of the speech recognition result.

[0075] In some embodiments, the received voice input includes one or more first speech segments received in a first input mode and one or more second speech segments received in a second input mode, and the voice recognition results include one or more first recognition results corresponding to the one or more first speech segments and one or more second recognition results corresponding to the one or more second speech segments, and process 600 also includes: based on the voice recognition results, using a language model to generate a response to the voice input, wherein the second recognition result in the voice recognition results is used to assist the language model in understanding the first recognition result adjacent to the second recognition result.

[0076] In some embodiments, the voice input is issued by the user to the virtual object during the interaction between the user and the virtual object, and one or more first recognition results include the user's dialogue content to the virtual object, and one or more second recognition results include explanation information of the dialogue content.

[0077] In some embodiments, using a language model to generate a response to a voice input includes: generating a first part of model prompt information based on one or more first recognition results, the first part being used to provide conversation content; generating a second part of model prompt information based on one or more second recognition results, the second part being used to provide explanatory information for the conversation content; providing the model prompt information to a language model to obtain an output of the language model; and determining a response based on the output of the language model.

[0078] Example devices and equipment

[0079] Figure 7 A schematic structural block diagram of an apparatus 700 for speech processing according to some embodiments of the present disclosure is shown. The apparatus 700 may be implemented as or included in a terminal device 110. Each module / component in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0080] like Figure 7 As shown, the device 700 includes a startup module 710, which is configured to start a first input mode to receive voice input in response to a first type of operation on a voice input area. The device 700 also includes a first switching module, which is configured to switch to a second input mode to receive voice input in response to a first operation of a second type on the voice input area in the first input mode. The device 700 also includes a result presentation module, which is configured to present a voice recognition result corresponding to the received voice input, in which the first recognition result corresponding to the voice received in the first input mode is distinguishable from the second recognition result corresponding to the voice received in the second mode.

[0081] In some embodiments, the second recognition result has a first delimiter and a second delimiter at the beginning and the end, respectively.

[0082] In some embodiments, the first recognition result includes text corresponding to speech received in the first input mode, and the second recognition result includes symbols matching speech received in the second input mode.

[0083] In some embodiments, the device 700 also includes a second switching module configured to switch to the first input mode to receive voice input in response to a second operation of a second type on the voice input area in the second input mode, wherein the second operation is opposite to the first operation direction.

[0084] In some embodiments, in a first input mode, the voice input area is presented in a first style, and in a second input mode, the voice input area is presented in a second style.

[0085] In some embodiments, the device 700 further includes a first motion effect module configured to present a first preset motion effect when the voice input area changes from the first style to the second style. The device 700 further includes a second motion effect module configured to present a second preset motion effect when the voice input area changes from the second style to the first style.

[0086] In some embodiments, the operation that triggers the switching between the first input mode and the second input mode is a sliding operation, and the presentation of the first preset animation effect or the second preset animation effect in response to the sliding operation satisfies at least one of the following: the sliding distance of the sliding operation is greater than or equal to the preset distance, or the sliding speed of the sliding operation is greater than or equal to the preset speed.

[0087] In some embodiments, the device 700 also includes a speech recognition module, which is configured to, in response to switching from a first input mode to a second input mode, provide a first speech received in the first input mode to a speech recognition model to obtain a first recognition result corresponding to the first speech; in response to switching from the second input mode to the first input mode, provide a second speech received in the second input mode to the speech recognition model to obtain a second recognition result corresponding to the second speech; and merge the first recognition result and the second recognition result as at least part of the speech recognition result.

[0088] In some embodiments, the received voice input includes one or more first voice segments received in a first input mode and one or more second voice segments received in a second input mode, and the voice recognition results include one or more first recognition results corresponding to the one or more first voice segments and one or more second recognition results corresponding to the one or more second voice segments, and the device 700 also includes a reply module, which is configured to generate a reply to the voice input based on the voice recognition result and using a language model, wherein the second recognition result in the voice recognition result is used to assist the language model in understanding the first recognition result adjacent to the second recognition result.

[0089] In some embodiments, the voice input is issued by the user to the virtual object during the interaction between the user and the virtual object, and one or more first recognition results include the user's dialogue content to the virtual object, and one or more second recognition results include explanation information of the dialogue content.

[0090] In some embodiments, the reply module is also configured to generate a first part of the model prompt information based on one or more first recognition results, and the first part is used to provide the content of the conversation; generate a second part of the model prompt information based on one or more second recognition results, and the second part is used to provide explanatory information for the content of the conversation; provide the model prompt information to the language model to obtain the output of the language model; and determine the reply based on the output of the language model.

[0091] The units and / or modules included in the device 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 700 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0092] Figure 8 8 is a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 8 The electronic device 800 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein.

[0093] like Figure 8As shown, the electronic device 800 is in the form of a general electronic device. The components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

[0094] The electronic device 800 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be a volatile memory (e.g., register, cache, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory) or some combination thereof. The storage device 830 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 800.

[0095] The electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 8 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0096] The communication unit 840 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0097] The input device 850 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 860 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 800 may also communicate with one or more external devices (not shown) through the communication unit 840 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 800, or communicate with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0098] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0099] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0101] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0102] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0103] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A speech processing method, comprising: In response to a first type of operation on the voice input area, initiating a first input mode to receive voice input; In the first input mode, in response to a first operation of a second type on the voice input area, switching to a second input mode to receive voice input; as well as Speech recognition results corresponding to the received speech input are presented, in which a first recognition result corresponding to speech received in the first input mode is distinguishable from a second recognition result corresponding to speech received in the second mode. 2 . The method according to claim 1 , wherein the second recognition result has a first delimiter and a second delimiter at the beginning and the end, respectively.

3. The method according to claim 1, wherein the first recognition result comprises text corresponding to the speech received in the first input mode, and The second recognition result includes symbols matched with the voice received in the second input mode.

4. The method according to claim 1, further comprising: In the second input mode, in response to a second operation of a second type on the voice input area, the mode is switched to the first input mode to receive voice input, wherein the second operation is in the opposite direction to the first operation.

5. The method according to claim 1, wherein in the first input mode, the voice input area is presented in a first style, and In the second input mode, the voice input area is presented in a second style.

6. The method according to claim 5, further comprising at least one of the following: During the process of the voice input area changing from the first style to the second style, a first preset animation effect is presented, or During the process of the voice input area changing from the second style to the first style, a second preset animation effect is presented.

7. The method according to claim 6, wherein the operation that triggers the switching between the first input mode and the second input mode is a sliding operation, and the presentation of the first preset animation effect or the second preset animation effect in response to the sliding operation satisfies at least one of the following: The sliding distance of the sliding operation is greater than or equal to a preset distance, or The sliding speed of the sliding operation is greater than or equal to a preset speed.

8. The method according to claim 1, wherein the speech recognition result is generated by: In response to switching from the first input mode to the second input mode, providing the first speech received in the first input mode to a speech recognition model to obtain the first recognition result corresponding to the first speech; In response to switching from the second input mode to the first input mode, providing a second speech received in the second input mode to the speech recognition model to obtain a second recognition result corresponding to the second speech; as well as The first recognition result and the second recognition result are combined as at least a part of the speech recognition result.

9. The method according to claim 1, wherein the received voice input comprises one or more first voice segments received in the first input mode and one or more second voice segments received in the second input mode, The speech recognition results include one or more first recognition results corresponding to the one or more first speech segments and one or more second recognition results corresponding to the one or more second speech segments, and The method further comprises: Based on the speech recognition result, a response to the speech input is generated using a language model, wherein a second recognition result in the speech recognition result is used to assist the language model in understanding a first recognition result adjacent to the second recognition result.

10. The method according to claim 9, wherein the voice input is issued to the virtual object during the interaction between the user and the virtual object, and the one or more first recognition results include the conversation content of the user to the virtual object, and the one or more second recognition results include explanation information of the conversation content.

11. The method of claim 9, wherein generating a response to the speech input using a language model comprises: Based on the one or more first recognition results, generate a first part of model prompt information, where the first part is used to provide dialogue content; Based on the one or more second recognition results, generating a second part of the model prompt information, wherein the second part is used to provide explanation information for the conversation content; Providing the model prompt information to the language model to obtain an output of the language model; as well as Based on the output of the language model, the response is determined.

12. A device for speech processing, comprising: A start module configured to start a first input mode to receive voice input in response to a first type of operation on the voice input area; A first switching module is configured to switch to a second input mode to receive voice input in response to a first operation of a second type on the voice input area in the first input mode; as well as A result presentation module is configured to present speech recognition results corresponding to received speech input, in which a first recognition result corresponding to the speech received in the first input mode is distinguishable from a second recognition result corresponding to the speech received in the second mode.

13. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 11.

Images