Voice interaction method and electronic equipment

Abstract

A voice interaction method and an electronic device are used for improving the accuracy and convenience of a voice input mode. According to the method, the electronic equipment provides selection entrances for a plurality of voice input methods on a display screen for a user to select the plurality of voice input methods which can be in one-to-one correspondence with a plurality of voice input modes, and the user can perform voice interaction with the electronic equipment in the corresponding voice input mode according to the voice input method selected by the user. Therefore, the two hands of the user are liberated, and the accuracy and convenience of voice input are improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202410805543.4, and the original application date is June 20, 2024. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the technical field of electronic equipment, and in particular to a voice interaction method and electronic equipment. Background Art

[0003] At present, voice input has become the main interaction method for large-screen devices such as foldable phones and tablets. However, due to the influence of homophones, user accents, or environmental noise, electronic devices often make conversion errors when recognizing user input voice and converting voice into corresponding text. In this case, users are required to manually edit to modify the erroneous text.

[0004] If voice input and manual correction of erroneous text are used in the above large-screen devices, the complexity of user operation will be increased. Similar problems also exist in small-screen devices with smaller screen sizes (such as smart wearable devices, etc.) or in-vehicle terminals where it is inconvenient for users to manually operate the screen.

[0005] Therefore, how to improve the accuracy and convenience of voice input is still an important issue that needs to be solved urgently. Summary of the invention

[0006] The present application provides a voice interaction method and an electronic device for improving the accuracy and convenience of voice input.

[0007] In the first aspect, the present application provides a voice interaction method, the execution subject of the method is an electronic device, the electronic device supports multiple voice input modes, and has a display screen. In the method, the electronic device provides multiple voice input method selection entrances on the display screen, and the multiple voice input methods correspond to the multiple voice input modes one by one; detects the user's selection operation of the multiple voice input methods; determines a first voice input method from the multiple voice input methods according to the selection operation, and the first voice input method corresponds to the first voice input mode in the multiple voice input modes; in the first voice input mode, performs a first operation according to the first voice information input by the user.

[0008] Through the above method, the electronic device can perform corresponding operations according to the voice information input by the user in the voice input mode selected by the user to realize voice interaction with the user. This method can effectively identify the user's voice input request and respond according to the user's request, thereby freeing the user's hands and improving the accuracy and convenience of the voice input method.

[0009] In one possible design, the multiple voice input modes include at least one of the following: voice-based direct input mode; voice-based text editing mode; voice-based text query mode; voice-based text creation mode; voice-based translation input mode; voice-based shooting input mode; voice-based traditional Chinese input mode; voice-based writing input mode; voice-based memory input mode; voice-based mode for obtaining cross-application target information; and voice-based mode for indicating cross-application target operations. It should be understood that this is only an example of the voice input mode that may be involved in the embodiments of the present application and is not a limitation. In other embodiments, in different business scenarios, the electronic device may also provide other voice input modes to the user, which will not be repeated here.

[0010] In one possible design, if the first voice input mode is a voice-based text editing mode, the first voice information describes the first character in at least one of the following ways: a commonly used character combination description method, a character decomposition description method, or a semantic description method; the first operation is performed according to the first voice information input by the user, including: replacing the second character displayed on the display screen with the first character, the first character and the second character are homophones or similar-sounding characters.

[0011] Through the above method, the user can use voice input to give character descriptions to the electronic device, so that the electronic device can perform text editing operations according to the user's voice descriptions and realize text editing, thereby freeing the user's hands and improving the accuracy and convenience of voice input.

[0012] In one possible design, if the first voice input mode is a mode for obtaining cross-application target information based on voice, in the first voice input mode, a first operation is performed according to the first voice information input by the user, including: in the first voice input mode, on a first interface provided by the electronic device, receiving the first voice information input by the user; obtaining target information from a second application according to the first voice information input by the user; and displaying the target information on the first interface.

[0013] Through the above method, users can control cross-applications through voice input mode. Users do not need to switch manually, which reduces user control operations and improves the accuracy and convenience of voice input.

[0014] In a possible design, providing multiple voice input method selection entrances on the display screen includes: providing a first button on the display screen; detecting a second operation of the user on the first button; and providing the multiple voice input method selection entrances on the display screen according to the second operation, wherein the first button is any one of a plurality of buttons, and the plurality of buttons correspond one-to-one to the multiple voice input methods. Exemplarily, the first button is a physical button; or, the first button is a virtual button.

[0015] The above method is used to illustrate the presentation of multiple selection entrances, such as direct display or display in response to user operation, and does not constitute any limitation.

[0016] In one possible design, the multiple voice input methods correspond one-to-one to multiple keys, and the multiple keys are physical keys or virtual keys.

[0017] In a possible design, selection entries for the multiple voice input methods are included in a function menu provided by the electronic device.

[0018] In a possible design, performing a first operation based on the first voice information input by the user includes: obtaining a semantic analysis result of the first voice information; and performing the first operation based on the semantic analysis result.

[0019] In one possible design, obtaining the semantic analysis result of the first voice information includes: sending a first request message to a server, wherein the first request message includes the first voice information; and receiving a first response message from the server, wherein the first response message includes the semantic analysis result of the first voice information.

[0020] In a possible design, a large-scale language model is deployed on the server, and the large-scale language model is used to perform semantic analysis on the first voice information to obtain the semantic analysis result.

[0021] In a possible design, performing the first operation based on the first voice information input by the user also includes: obtaining first text information, the first text information including a voice recognition result of the first voice information; and displaying the first text information on the display screen.

[0022] In one possible design, obtaining the first text information includes: sending a second request message to a server, the second request message including the first voice information; receiving a second response message from the server, the second response message including the first text information obtained by recognizing the first voice information.

[0023] In a second aspect, an electronic device is provided, the device comprising a plurality of functional modules; the plurality of functional modules interact with each other to implement the method performed by the first electronic device in the first aspect and its respective embodiments. The plurality of functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the plurality of functional modules can be arbitrarily combined or divided based on specific implementation.

[0024] The third aspect provides a device, comprising at least one processor and at least one memory, wherein the at least one memory stores computer program instructions, and when the device is running, the at least one processor executes the method executed by the first electronic device in the above-mentioned first aspect and each embodiment thereof.

[0025] The fourth aspect also provides a program product, which, when executed on a device, enables the device to execute the method executed by the first electronic device in any one of the above aspects and its various embodiments.

[0026] The fifth aspect also provides a readable storage medium, in which a program is stored. When the program is executed by a device, the device executes the method executed by the first electronic device in any of the above aspects and its various embodiments.

[0027] The sixth aspect also provides a chip, which is used to read the program stored in the memory and execute the method executed by the first electronic device in any of the above aspects and its various embodiments.

[0028] A seventh aspect further provides a chip system, the chip system including a processor, for supporting a device to implement the method executed by the first electronic device in any of the above aspects and each embodiment thereof. In one possible design, the chip system also includes a memory, the memory being used to store the necessary programs and data. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0029] It should be noted that for the beneficial effects of various designs of the electronic devices provided in the second to seventh aspects of the embodiments of the present application, please refer to the beneficial effects of any possible design in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a manual editing method for a large-screen device is shown;

[0031] Figure 2 A schematic diagram of the hardware structure of a possible electronic device is shown;

[0032] Figure 3 A software architecture block diagram of an electronic device provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a system architecture that may be applicable to a voice interaction method provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of a scenario in which a voice interaction method provided in an embodiment of the present application may be applicable;

[0035] Figure 6a A schematic diagram of another scenario to which a voice interaction method provided in an embodiment of the present application may be applicable;

[0036] Figure 6b A schematic diagram of another scenario to which a voice interaction method provided in an embodiment of the present application may be applicable;

[0037] Figure 7 A schematic diagram of another scenario to which a voice interaction method provided in an embodiment of the present application may be applicable;

[0038] Figure 8 A schematic diagram of another scenario to which a voice interaction method provided in an embodiment of the present application may be applicable;

[0039] Fig. 9 A schematic diagram of another scenario to which a voice interaction method provided in an embodiment of the present application may be applicable;

[0040] Fig.10 A schematic diagram of another scenario to which a voice interaction method provided in an embodiment of the present application may be applicable;

[0041] Figure 11-13 A schematic diagram of a method for providing selection entrances for multiple voice input methods provided in an embodiment of the present application;

[0042] Figure 14-18 A flowchart of different examples of a voice interaction method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and embodiments.

[0044] At present, electronic devices are used more and more frequently in people's work and life. Among them, electronic devices with smaller screen sizes are usually called small-screen devices, such as smart wearable watches, etc. Electronic devices with larger screen sizes are usually called large-screen devices, such as foldable screen mobile phones, tablets, etc.

[0045] In order to provide users with a better immersive experience, the sizes of display screens of some electronic devices are gradually increasing. At the same time, the manual controllability of users over electronic devices is reduced.

[0046] like Figure 1As shown, taking a foldable screen mobile phone as an example, when the foldable screen of the foldable screen mobile phone is placed in an unfolded state along the folding axis of the foldable screen mobile phone, more text (such as "XXX", "YYY", "ZZZ", etc.) can be presented on the foldable screen of the mobile phone to facilitate user reading, thereby improving the user's reading experience. However, when the foldable screen is in the unfolded state, the horizontal screen size of the mobile phone is wider. When the text at a certain position on the screen needs to be edited, the keyboard will be divided into two parts and set on both sides of the screen. The user needs to use both hands to manipulate the buttons on the left and right parts of the keyboard respectively to manually input text or modify the original text, which brings great inconvenience to the user. Schematically, Figure 1 The text "ZZZZZZZZZZ" at the marked position is replaced with the text corresponding to the pinyin "kao'lv". As the user clicks the pinyin k, a, o, l, v in the corresponding keyboard in sequence, multiple optional texts are presented on the foldable screen, and the user can click "Consider" as the replaced text content.

[0047] In view of the problems existing in the manual input method in the above electronic devices, voice input (or voice-controlled input) has gradually become an important interaction method.

[0048] However, in the scenario where voice input is used, the electronic device also needs to recognize the voice information input by the user, and convert the voice information into corresponding text information and display it. Affected by homophones, user accents, or environmental noise, electronic devices often make conversion errors when recognizing and converting the voice input by the user into text. At this time, the user is also required to manually edit to modify the erroneous text. If voice input and manual modification of erroneous text are used in the above-mentioned large-screen devices, the complexity of user operation will be increased. In scenarios where it is inconvenient for users to manually operate the screen, such as small-screen devices with smaller screen sizes (such as smart wearable devices, etc.) or in-vehicle terminals, similar problems also exist when voice input and manual modification of erroneous text are used.

[0049] Therefore, how to improve the accuracy and convenience of voice input remains an important issue that needs to be solved urgently.

[0050] In view of the above problems, the embodiments of the present application provide a voice interaction method and an electronic device. In the method, the electronic device can provide a plurality of voice input methods for the user to choose from, and the plurality of voice input methods can correspond to a plurality of voice input modes one by one. According to the voice input method selected by the user, the user can perform voice interaction with the electronic device in the corresponding voice input mode to realize intelligent control of the electronic device, thereby freeing the user's hands and improving the accuracy and convenience of voice input.

[0051] The technical solutions in the embodiments of the present application can be applied to electronic devices, and the electronic devices can be any devices with a display screen or an associated display screen. For example, the electronic devices can be electronic devices such as mobile phones, folding screen mobile phones, tablet computers, wearable devices (e.g., watches, bracelets, glasses, etc.), vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), smart home devices (e.g., smart TVs, etc.). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0052] The electronic device to which the embodiments of the present application can be applied can also be a portable terminal device that also includes other functions such as a personal digital assistant and / or a music player. Exemplary embodiments of portable terminal devices include but are not limited to Or other electronic devices with operating systems.

[0053] Figure 2 : A schematic diagram of the hardware structure of a possible electronic device is shown. The electronic device 200 includes: a radio frequency (RF) circuit 210, a power supply 220, a processor 230, a memory 240, an input unit 250, a display unit 260, an audio circuit 270, a communication interface 280, and a wireless fidelity (Wi-Fi) module 290 and other components. Those skilled in the art can understand that Figure 2 The hardware structure of the electronic device 200 shown in the figure does not constitute a limitation on the electronic device 200. The electronic device 200 provided in the embodiment of the present application may include more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. Figure 2 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0054] Combine the following Figure 2 The components of the electronic device 200 are described in detail:

[0055] The RF circuit 210 can be used for receiving and sending data during communication or calls. In particular, after receiving downlink data from the base station, the RF circuit 210 sends it to the processor 230 for processing; in addition, the uplink data to be sent is sent to the base station. Generally, the RF circuit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc.

[0056] In addition, the RF circuit 210 can also communicate with other devices through a wireless communication network. The wireless communication can use any communication standard or protocol, including but not limited to global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0057] Wi-Fi technology is a short-range wireless transmission technology. The electronic device 200 can connect to an access point (AP) through the Wi-Fi module 290 to access the data network. The Wi-Fi module 290 can be used for receiving and sending data during the communication process.

[0058] The electronic device 200 can be physically connected to other devices through the communication interface 280. Optionally, the communication interface 280 is connected to the communication interface of the other device through a cable to achieve data transmission between the electronic device 200 and the other device.

[0059] The electronic device 200 can also implement communication services and interact with other electronic devices, so the electronic device 200 needs to have a data transmission function, that is, the electronic device 200 needs to include a communication module. Figure 2 Communication modules such as the RF circuit 210, the Wi-Fi module 290, and the communication interface 280 are shown, but it is understandable that at least one of the above components or other communication modules (such as a Bluetooth module) for implementing communication exist in the electronic device 200 to perform data transmission.

[0060] For example, when the electronic device 200 is a mobile phone, the electronic device 200 may include the RF circuit 210, and may also include the Wi-Fi module 290, or may include a Bluetooth module ( Figure 2 When the electronic device 200 is a tablet computer, the electronic device 200 may include the Wi-Fi module, or may include a Bluetooth module ( Figure 2 When the electronic device 200 is a smart home device, the electronic device 200 may include the Wi-Fi module 290, or may include a Bluetooth module ( Figure 2 not shown).

[0061] The memory 240 can be used to store software programs and modules. The processor 230 executes various functional applications and data processing of the electronic device 200 by running the software programs and modules stored in the memory 240. Optionally, the memory 240 may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system (mainly including software programs or modules corresponding to the kernel layer, system layer, application framework layer, and application layer, etc.).

[0062] In addition, the memory 240 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0063] The input unit 250 can be used to receive the editing operations of various types of data objects such as digital or character information input by the user, and generate key signal input related to the user settings and function control of the electronic device 200. Optionally, the input unit 250 can include a touch panel 251 and other input devices 252.

[0064] Among them, the touch panel 251, also known as a touch screen, can collect the user's touch operations on or near it (such as the user's operation on or near the touch panel 251 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. In an embodiment of the present application, the touch panel 251 can collect the user's user operations on it. For example, the user operation may include the operation of the user clicking a physical button on the touch panel 251, or the user operation may also include the operation of the user long pressing a physical button on the touch panel 251 to switch the physical button to an operation of a physical button with other functions.

[0065] Optionally, the other input device 252 may include, but is not limited to, one or more of a physical keyboard, an infrared sensor, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, a joystick, etc. For example, an infrared sensor may be used to obtain a user's air gesture operation.

[0066] The display unit 260 can be used to display information input by the user or information provided to the user and various menus of the electronic device 200. The display unit 260 is the display system of the electronic device 200, which is used to present an interface and realize human-computer interaction. The display unit 260 may include a display panel 261. Optionally, the display panel 261 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. In an embodiment of the present application, the display unit 260 can be used to display a user interface. For example, the user interface can be the desktop (or main interface) of the electronic device, or it can be an interface corresponding to various possible business scenarios, such as a chat interface of an instant messaging APP, a memo interface, a multimedia video APP interface, etc.

[0067] The processor 230 is the control center of the electronic device 200, and uses various interfaces and lines to connect various components, and executes various functions and processes data of the electronic device 200 by running or executing software programs and / or modules stored in the memory 240, and calling data stored in the memory 240, so as to realize various services based on the electronic device 200. In the embodiment of the present application, the processor 230 can be used to implement a voice interaction method provided in the embodiment of the present application.

[0068] The electronic device 200 also includes a power supply 220 (such as a battery) for supplying power to various components. Optionally, the power supply 220 can be logically connected to the processor 230 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0069] like Figure 2As shown, the electronic device 200 also includes an audio circuit 270, a microphone 271 and a speaker 272, which can provide an audio interface between the user and the electronic device 200. The audio circuit 270 can be used to convert audio data into a signal that can be recognized by the speaker 272, and transmit the signal to the speaker 272, which is converted into a sound signal and output by the speaker 272. The microphone 271 is used to collect external sound signals (such as the sound of a person speaking, or other sounds, etc.), and convert the collected external sound signals into signals that can be recognized by the audio circuit 270, and send them to the audio circuit 270. The audio circuit 270 can also be used to convert the signal sent by the microphone 271 into audio data, and then output the audio data to the RF circuit 210 to send it to, for example, another electronic device, or output the audio data to the memory 240 for subsequent further processing.

[0070] although Figure 2 Not shown, the electronic device 200 may also include a camera, at least one sensor, etc., which will not be described in detail here. The at least one sensor may include but is not limited to a pressure sensor, an air pressure sensor, an acceleration sensor, a distance sensor, a fingerprint sensor, a touch sensor, a temperature sensor, etc.

[0071] Figure 3 The following is a software architecture diagram of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the software structure of the electronic device can be a layered architecture, for example, the software can be divided into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer (framework, FWK), the runtime (runtime) and the system library, and the kernel layer.

[0072] The application layer can include a series of application packages. Figure 3 As shown, the application layer may include camera, settings, skin module, user interface (UI), third-party applications, etc. Among them, the third-party applications may include gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. In the embodiment of the present application, the application layer may include a target installation package of a target application that the electronic device requests to download from a server, and the function files and layout files in the target installation package are adapted to the electronic device.

[0073] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer may include some predefined functions. Figure 3 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, and a notification manager.

[0074] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0075] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0076] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0077] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0078] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0079] The runtime includes the core library and the virtual machine. The runtime is responsible for the scheduling and management of the operating system.

[0080] The core library consists of two parts: one is the function that the computer programming language needs to call, and the other is the core library of the operating system. The application layer and the application framework layer run in the virtual machine. Taking the Java language as a computer programming language as an example, the virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0081] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), image processing library, etc.

[0082] The surface manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0083] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0084] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0085] The two-dimensional graphics engine is a drawing engine for two-dimensional drawing. The two-dimensional graphics engine can perform drawing operations, such as drawing different voice buttons or drawing different forms of voice buttons on the screen of the electronic device to receive different voice information input by the user in different voice input method scenarios. Alternatively, the two-dimensional graphics engine can also draw a menu on the display screen of the electronic device, which is a selection entry for multiple voice input methods, for the user to select the voice input method desired to use from the menu, so as to provide the user with corresponding functions.

[0086] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0087] The hardware layer can include various types of sensors, such as accelerometers, gyroscopes, touch sensors, etc.

[0088] It should be noted that Figure 2 and Figure 3The structure shown is only an example of the electronic device provided by the embodiment of the present application, and cannot be used to limit the electronic device provided by the embodiment of the present application. In specific implementation, the electronic device may have more Figure 2 or Figure 3 More or fewer devices or modules may be included in the structures shown.

[0089] Usually, the electronic device 200 can run multiple applications at the same time. In a simpler case, one application can correspond to one process, and in a more complex case, one application can correspond to multiple processes. Each process has a process number (process ID).

[0090] It should be understood that in the embodiments of the present application, "at least one of the following" or similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple. "Multiple" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0091] In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order. For example, the "first voice input method" and "second voice input method" in the following embodiments are only used to distinguish different voice input methods, and are not used to limit the specific implementation methods of the voice input methods.

[0092] It should be understood that the hardware structure of the electronic device can be as follows Figure 2 As shown, the software system architecture can be Figure 3 As shown, the software programs and / or modules corresponding to the software system architecture in the electronic device can be stored in the memory 240, and the processor 230 can run the software programs and applications stored in the memory 240 to execute the process of a voice interaction method provided in an embodiment of the present application.

[0093] Figure 4 A schematic diagram of a system architecture to which the voice interaction method provided in an embodiment of the present application may be applicable is shown.

[0094] like Figure 4 As shown in A in FIG. 1 , the system architecture may include an electronic device 200 , a user 41 , and a server 42 .

[0095] The electronic device 200 may receive voice information input by the user 41. The electronic device 200 may send a request message to the server 42 through the communication network according to the voice information, requesting the server 42 to perform voice recognition, semantic analysis (or semantic understanding) and other processing on the voice information.

[0096] For example, an automatic speech recognition (ASR) module may be deployed in the server 42, and the ASR module may perform speech recognition on the speech information from the electronic device 200, and convert the speech information into corresponding text information, that is, obtain speech recognition results. Alternatively, a large language model (large language model) (referred to as a large model) may be deployed in the server 42, and the large model may perform semantic analysis on the speech information from the electronic device 200 to obtain corresponding semantic analysis results.

[0097] The server 42 can send the speech recognition result or the semantic analysis result as the response information to the above request message to the electronic device 200. Accordingly, the electronic device 200 can receive the response information from the server 42, display the received response information through the display screen, and / or perform corresponding operations based on the response information.

[0098] In another possible implementation, Figure 4 As shown in B in the figure, the voice interaction method can also be implemented by the electronic device 200 itself in the embodiment of the present application. For example, an ASR module can be deployed in the electronic device 200, and the ASR module can be used to perform voice recognition on the voice information input by the user 41 to obtain converted text information. Alternatively, a large model can be deployed in the electronic device 200, and the large model can perform semantic analysis on the voice information input by the user 41 to obtain corresponding semantic analysis results. The electronic device 200 can display the voice recognition result or the semantic analysis result on its display screen, and can also perform corresponding operations according to the voice recognition result or the semantic analysis result.

[0099] It should be understood that in the embodiments of the present application, Figure 4In the situation shown in A, the ASR module and the large model can be deployed on the same server or on different servers. The electronic device can directly interact with the corresponding server, or it can be forwarded by other devices to indirectly interact with the corresponding server. For example, the electronic device can send a request message containing voice information to the application server of the currently running application. The application server can forward the received request message to the server deployed with the large model. The server deployed with the large model performs semantic analysis on the received voice information and returns the semantic understanding result to the application server. The application server can encapsulate the semantic analysis result and feed it back to the electronic device. The embodiments of the present application are not limited to this.

[0100] based on Figure 2-Figure 3 The structure diagram of the introduction and Figure 4 The following is a schematic diagram of the system architecture, taking a mobile phone as an example of an electronic device to introduce several possible voice interaction scenarios involved in the embodiments of the present application.

[0101] In the embodiment of the present application, the electronic device can provide a user with a selection entry of multiple voice input methods on the display screen for the user to choose, and the multiple voice input methods correspond to the multiple voice input modes. If the user's selection operation of a certain voice input method is detected, the electronic device can perform a corresponding operation according to the voice information input by the user in the corresponding voice input mode.

[0102] As an example, the multiple voice input modes may include at least one of the following: voice-based direct input mode; voice-based text editing mode; voice-based text query mode; voice-based text creation mode; voice-based translation input mode; voice-based shooting input mode; voice-based traditional Chinese input mode; voice-based writing assistance input mode; voice-based memory input mode; voice-based mode for obtaining cross-application target information; and voice-based mode for indicating cross-application target operations.

[0103] Among them, the voice-based direct input mode is a traditional voice input mode. The user can select the voice-based direct input mode. In the voice-based direct input mode, the user can input various types of voice information into the electronic device, and the electronic device can perform voice recognition and display the voice information input by the user.

[0104] The voice-based text editing mode is a mode for implementing editing operations on text content or text format based on voice input. The user can select the voice-based text editing mode. In the voice-based text editing mode, the electronic device can edit the text information displayed on the display screen according to the voice information input by the user, and obtain the edited text information. The text information displayed on the display screen here can be the text information displayed on the interface of the reading application or other applications, or it can be the text information obtained after the voice information input by the user is recognized. The embodiment of the present application does not limit the acquisition method of the modified text information. The editing operation of the text content may include but is not limited to at least one of the operations of adding, deleting or modifying the text content. The editing of the text format may include text line breaks, first line indentation, etc. The embodiment of the present application does not specifically limit the editing operations involved in the voice-based text editing mode.

[0105] The voice-based text query mode is a mode for querying text information based on voice input. The user can select the voice-based text query mode. In the voice-based text query mode, the electronic device can query the number of words (or characters) of the text information based on the voice information input by the user, or the location of a certain word or character in the text information, or the frequency of occurrence of a certain word or character, etc. The embodiment of the present application does not specifically limit this query operation.

[0106] The voice-based text creation mode is a mode for creating and processing text information based on voice input. Users can choose the voice-based text creation mode. In the voice-based text creation mode, users can provide creation requirements to the electronic device through voice input, and the electronic device can perform text creation or polishing and rewriting according to the user's requirements to obtain text information that meets the user's demands. Here, when the electronic device performs text creation or polishing and rewriting, it can be implemented by using the auxiliary functions of an application itself, or it can be implemented in a cross-application manner through an application with artificial intelligence functions, and the embodiments of the present application do not limit this.

[0107] The voice-based translation input mode is a mode that implements text translation processing based on voice input. Users can select the voice-based translation input mode. In the voice-based translation input mode, users can input voice information into an electronic device in one language, and the electronic device can recognize the voice information, translate it into text information in another language and display it. The voice-based translation input mode can include but is not limited to translation between any two languages ​​such as Chinese, English, Japanese, Korean, German, etc.

[0108] The voice-based shooting input mode is a mode that implements shooting input based on voice input. Users can select the voice-based shooting input mode. In the voice-based shooting input mode, when the user uses the camera of the electronic device to point to an image, the user can indicate the text to be shot through voice information so that the electronic device can extract the target text from the shot image.

[0109] The voice-based traditional Chinese input mode is a mode for implementing traditional Chinese input based on voice input. Users can select the voice-based traditional Chinese input mode. In the voice-based traditional Chinese input mode, the electronic device can recognize the voice information input by the user and convert the recognized text information into traditional Chinese characters for display.

[0110] The voice-based writing help input mode is a mode that implements the writing help input function based on voice input. Users can select the voice-based writing help input mode. In the voice-based writing help input mode, the electronic device can help the user write text, such as greetings, birthday wishes, holiday wishes, etc. More intelligently, the electronic device can also help the user write essays, papers, or business reports for user reference. The embodiment of the present application does not specifically limit the writing help input mode.

[0111] The voice-based memory input mode is a mode for acquiring memory information based on voice input. The user can select the voice-based memory input mode. In the voice-based memory input mode, the user can describe the memory information to be acquired by voice, such as user ID number information, residential address information, contact number, etc. The electronic device acquires the corresponding memory information based on the voice information input by the user. Here, the memory information can be information collected in the current application, or it can be information recorded in other applications across applications, such as recorded in the "Xiaoyi Memory" application, which is not limited in the embodiments of the present application.

[0112] The mode of obtaining cross-application target information based on voice is a mode of obtaining target information across applications based on voice input. The user can select the mode of obtaining cross-application target information based on voice. In this mode, the user can obtain target information across applications in the current application, for example, obtaining weather information of a certain area from a weather application, obtaining music, video, short video and other resources from multimedia applications, obtaining address information or phone numbers of hotels near scenic spots from travel applications, obtaining takeout phone numbers from takeout applications, etc. The embodiments of the present application do not limit this.

[0113] The mode of cross-application target operation based on voice instruction is a mode of instructing the electronic device to perform cross-application target operation based on voice input. The user can select the mode of cross-application target operation based on voice instruction. In this mode, the user can input voice information to the electronic device in the interface of the current application, and the voice information indicates the target application and the target operation, such as playing the target music or video in multimedia applications, and booking a hotel near the target attraction in travel applications.

[0114] For ease of understanding, the following describes voice interaction scenarios involving various voice input modes with examples in conjunction with the accompanying drawings.

[0115] Voice interaction scenario A:

[0116] In a voice interaction scenario, a user may select a voice-based direct input mode to perform voice interaction with an electronic device.

[0117] See also Figure 5 FIG. 1 is a schematic diagram of a scenario in which a voice interaction method provided in an embodiment of the present application may be applicable. Figure 5 Interface 501 in the figure shows a schematic diagram of a memo of a mobile phone. In interface 501, on the newly added memo interface, the electronic device may provide the user with a selection entry for a voice input method (for example, represented as voice input method 1), and the selection entry for voice input method 1 may be represented as button 1, for example. The user may select the voice input method 1 by selecting button 1. For example, the user may click button 1 in the interface 501 to select input method 1. Voice input method 1 may correspond to voice input mode 1, and the voice input mode 1 may be, for example, a direct input mode based on voice. In voice input mode 1, the user may input voice to the electronic device, for example, represented as voice 1. As an example, the voice 1 may be a voice inputting the following recipe to the memo of the electronic device: "Sauté chopped onion, green onion slices, celery shreds, mushroom slices, capers, black olives, and cherry tomatoes with olive oil. Add butter, sea salt, and black pepper". The electronic device may perform voice recognition on the content of voice 1, and add and display the corresponding voice recognition result in the memo of interface 501, that is, text 1 corresponding to voice 1.

[0118] The electronic device may provide the user with a selection entry for another voice input method (e.g., represented as voice input method 2), and the selection entry for voice input method 2 may be represented, for example, as button 2 in interface 502, interface 503, or interface 504. If the user wishes to edit the content or format of text 1, the user may select voice input method 2 by selecting button 2. For example, the user may click button 2 in the interface 502 to select input method 2. Voice input method 2 may correspond to voice input mode 2, which may be, for example, a voice-based text editing mode. In voice input mode 2, the user may input voice information into the electronic device, for example, represented as voice 2. Voice 2 may be used to perform editing operations on the content or format of text 1, such as text content modification operations in interface 502, text line break operations in interface 503, and text deletion operations in interface 504.

[0119] Among them, as shown in interface 502, in the voice-based text editing mode, voice 2 can indicate the modification of the content in text 1. For example, voice 2 is the voice corresponding to the following text: "Change black pepper to white peppercorns". The electronic device can perform voice recognition on voice 2 and display the corresponding voice recognition result in the echo area next to button 2, that is, text 2 corresponding to voice 2. At the same time, the electronic device can also obtain the semantic analysis result of voice 2, and know that it is necessary to change the "black pepper" in text 1 recorded in the memo to "white peppercorns". The electronic device can also perform corresponding modification operations according to the semantic analysis result corresponding to voice 2. As a result, the text content of the recipe recorded in the mobile phone memo is changed to the following content "Fry chopped onion, green onion slices, celery shreds, mushroom slices, capers, black olives, and cherry tomatoes with olive oil. Add butter, sea salt, and white peppercorns", as shown in text 3-1 in interface 502.

[0120] As shown in interface 503, in the voice-based text editing mode, voice 2 can indicate the setting or modification of the format of the content in text 1. For example, voice 2 is the voice corresponding to the following text: "Start a new line starting with adding cream". The electronic device can perform voice recognition on voice 2 and display the corresponding voice recognition result, that is, text 2 corresponding to voice 2, in the echo area next to button 2. At the same time, the electronic device can also obtain the semantic analysis result of voice 2, and know that it is necessary to perform a line break operation on the text starting from "add cream" in text 1 recorded in the memo. The edited text is shown as text 3-2 in interface 503.

[0121] As shown in interface 504, in the voice-based text editing mode, voice 2 can indicate the deletion of the content in text 1. For example, voice 2 is the voice corresponding to the following text: "Delete black pepper". The electronic device can perform voice recognition on voice 2 and display the corresponding voice recognition result, that is, text 2 corresponding to voice 2, in the echo area next to button 2. At the same time, the electronic device can also obtain the semantic analysis result of voice 2, and know that the "black pepper" in text 1 recorded in the memo needs to be deleted. The electronic device can also perform corresponding deletion operations according to the semantic analysis result corresponding to voice 2. As a result, the text content of the recipe recorded in the mobile phone memo is changed to the following content "Fry chopped onion, green onion slices, celery shreds, mushroom slices, capers, black olives, and cherry tomatoes with olive oil. Add butter and sea salt", as shown in text 3-3 in interface 504.

[0122] It should be noted that the specific content and display form of interfaces 501 to 504 are not limited in the embodiments of the present application. For example, button 1 and / or button 2 can be located at the bottom of the interface or at the top of the interface, etc.; button 1 and / or button 2 can be displayed separately on the interface or in a function menu on the interface. The function menu is a selection entry for multiple buttons, which will not be described in detail here. In other embodiments, if the voice recognition of voice 1 is incorrect, resulting in erroneous text in text 1, the user can also use the method shown in interface 502 or interface 504 to input voice 2 into the electronic device in a voice-based text editing mode to edit the erroneous text. The implementation details can be found in the relevant description of voice interaction scenario C below, which will not be described here.

[0123] Voice interaction scenario B:

[0124] In a voice interaction scenario, a user may select a voice-based text query mode to perform voice interaction with an electronic device. In the voice-based text query mode, the electronic device may perform query operations based on the voice information input by the user. The query operations may include but are not limited to querying the number of words (or characters) in the text information, the location of a certain word or character in the text information, or the frequency of occurrence of a certain word or character. Figure 6a FIG. 1 is another schematic diagram of a scenario to which a voice interaction method provided in an embodiment of the present application may be applicable. Figure 6aInterface 601 in the figure shows a schematic diagram of the interface of a reading application (APP) for a mobile phone. In interface 601, the electronic device can display the content of the novel to the user, represented as text 1, schematically represented by "XXX", "YYY", "ZZZ", etc. At the same time, the electronic device can provide a selection entry for voice input method 2, such as button 2. The user clicks button 2 to select voice input method 2, and accordingly enters voice input mode 2 shown in interface 602, which can be, for example, a voice-based text query mode. In voice input mode 2, the user can input voice 2 to the electronic device to perform query operations on text 1 on the current interface.

[0125] As an example, in a voice-based text query mode, voice 2 may be, for example, a voice corresponding to the following text: "Query the number of words (or characters) in the text on this page", which is used to query the number of words (or characters) in the text on the current display interface. Accordingly, the electronic device may perform voice recognition on voice 2, and display the corresponding voice recognition result, that is, text 2 corresponding to voice 2, in the echo area next to button 2. At the same time, the electronic device may also obtain the semantic analysis result of voice 2, and know that it is necessary to query / count the number of words (or characters) in the text on the current interface. The electronic device performs the corresponding word count operation based on the semantic analysis result of voice 2, and displays the query result at the bottom of interface 602, for example, represented as text 3, which is used to indicate the number of words (or characters) in the text of the current interface: 44.

[0126] Similarly, if the user needs to query a certain text (or character) in interface 601, or query the location of a certain text or character, the user can click in interface 601 to select voice input method 2, and accordingly, enter the voice input mode 2 shown in interface 603, which can be, for example, a voice-based text query mode. In voice input mode 2, the user can input voice 2 to the electronic device to perform query operations on text 1 on the current interface. As an example, voice 2 can be the voice corresponding to the following text: "Query XX", which is used to query whether XX exists in text 1, or query the location of XX in text 1. Correspondingly, the electronic device can perform voice recognition on voice 2, and display the corresponding voice recognition result, that is, text 2 corresponding to voice 2, in the echo area next to button 2. At the same time, the electronic device can also obtain the semantic analysis result of voice 2, and know that XX needs to be queried on the current interface. The electronic device performs the corresponding query operation according to the semantic analysis result of voice 2, and highlights the queried text XX in the text 1 in interface 603.

[0127] Voice interaction scenario C:

[0128] In a voice interaction scenario, a user may select a voice-based text creation mode to perform voice interaction with an electronic device.

[0129] participate Figure 6b FIG. 1 is another schematic diagram of a scenario to which a voice interaction method provided in an embodiment of the present application may be applicable. Figure 6b Interface 604 in the figure shows a schematic diagram of a chat interface of an instant messaging APP on a mobile phone. In interface 604, a user can select voice input method 1 by clicking button 1. Voice input method 1 corresponds to voice input mode 1, which can be a direct input mode based on voice. In the direct input mode based on voice, a user can input voice 1 to the electronic device, specifically the voice corresponding to the following text: "Happy birthday, little B". The electronic device can recognize the voice 1 input by the user and insert the recognized text 1 into the input box of the chat interface.

[0130] The electronic device can also press button 2 shown in interface 605, which is associated with voice input method 2, corresponding to voice input mode 2, and the voice input mode 2 can be a voice-based text creation mode. If the user thinks that the text in interface 604 is relatively thin and insufficient to express user A's blessing to user B. In interface 605, the user can click button 2 to select voice input method 2, and voice input method 2 corresponds to voice input mode 2, which can be a voice-based text creation mode. In the voice-based text creation mode, the user can input voice 2 to the electronic device, specifically the voice corresponding to the following text: "Rewrite it for me". The electronic device can recognize the voice 2 input by the user and display the corresponding text 2 in the echo area. The electronic device can use a large-scale language model to perform semantic analysis on the voice 2. When it is known that the user has a request to rewrite the text 1 in the interface 604, the electronic device can use the large-scale language model to perform a polishing and rewriting operation on the text 1 to obtain text 3, as shown below: "Dear little B, happy birthday to you! May your smile be as bright as the sun and your life be as beautiful as poetry. Every year of growth is a new beginning. May you be healthy, happy, safe and successful in the days to come!" Only part of the content of text 3 is displayed in the input box of interface 605.

[0131] Similarly, the user may not provide any basic text and directly provide the user with a text request in the voice-based text creation mode in the interface 605. The electronic device may create text 3 for the user based on the voice information input by the user and use tools such as large-scale language models. This will not be repeated here.

[0132] Voice interaction scenario D:

[0133] See also Figure 7As shown in the figure, it is another schematic diagram of a scenario where a voice interaction method provided by an embodiment of the present application may be applicable. Figure 7 The interface 701 in Figure 7 shows a schematic diagram of the interface of the phone call APP on the mobile phone. In the interface 701, the electronic device can provide the user with historical communication records, a dialing keyboard, and a button 1. The button 1 is associated with a voice input method 1 and a voice input mode 1. The user can click the button 1 to select the voice input method 1 and enter the interface 702. The voice input mode 1 corresponding to the voice input method 1 is, for example, a direct voice input mode. In the interface 702, in the voice input mode 1, the user can input the following voice 1 corresponding to the text: "Call XXX" to enable the phone call APP on the mobile phone to call the target user "XXX". The electronic device can perform voice recognition on the voice 1 input by the user and display the text 1 corresponding to the voice 1 in a certain display area on the interface 702.

[0134] Taking the Chinese language scenario as an example, taking the voice pronunciation of the target user "XXX" as the following pinyin "wang jian yu" as an example, due to homophones, when the electronic device performs voice recognition on the voice 1, it wrongly recognizes the voice pronunciation "wang jian yu" of the target user as "Wang Jianyu" shown in the interface 702. Therefore, the wrong text needs to be modified.

[0135] In the interface 703, the electronic device can also provide the user with Button 2, and Button 2 is associated with the voice input method 2 and the voice input mode 2. The user can click Button 2 in the interface 703 to switch to the voice input method 2. The voice input mode 2 corresponding to the voice input method 2 can be, for example, a voice-based text editing mode. In the voice input mode 2, the user can input voice 2 to the electronic device to input a supplementary explanation or description of the voice 1 to the electronic device. Among them, for the homophonic characters to be replaced, voice 2 can describe the homophonic characters in at least one of the following ways: the common character combination description method; the character disassembling description method; the semantic description method. For example, in the interface 703, voice 2 can include the voice corresponding to the following text: "Jian is the character '健' with a single-person radical, and yu is the character '豫' of Henan". Among them, "Jian is the character '健' with a single-person radical" describes the homophonic character "jian" that is expected to be used in the character disassembling description method, and "yu is the character '豫' of Henan" describes the homophonic character "yu" that is expected to be used in the semantic description method. Optionally, using the common character combination description method, voice 2 can include the voice corresponding to the following text: "Jian is the '健' of '健康' (healthy), and yu is the '豫' of '毫不犹豫' (unhesitating)". The electronic device can perform voice recognition on the voice 2 input by the user and display the text 2 corresponding to the voice 2 in the echo area next to Button 2, as shown in the interface 703. At the same time, the electronic device can obtain the semantic analysis result of the voice 2 and perform corresponding operations according to the semantic analysis result. For example, modify the incorrect text "建" in the interface 702 to the homophone "健", and modify the incorrect text "玉" to the homophone "豫", as shown in the interface 704.

[0136] It should be noted that Figure 7 only the Chinese language scenario is taken as an example for illustration, and it does not constitute a limitation on the language scenario. In other language scenarios, such as English, German, French, etc., similar methods can be used to input voice information to the electronic device in different voice input modes. The electronic device can perform corresponding operations according to the recognition result and / or semantic analysis result of the voice information, which will not be elaborated here. Figure 7

[0137] The above voice interaction scenario C can also be implemented as a cross-application voice interaction scenario. For example Figure 7 the interface of the phone APP in can be replaced with the mobile phone desktop or the operation interface of other applications. On the mobile phone desktop or the operation interface of other applications, the user can perform cross-application operations in a similar way as Figure 7 to make a call to the target user. During the voice interaction process between the user and the electronic device, in the same way as Figure 7 shown, by selecting different voice input methods, different voice information can be input to the electronic device to achieve the purpose of making a call to the target user and improve the accuracy and convenience of voice input.​

[0138] Voice interaction scenario E:

[0139] See also Figure 8 FIG. 1 is another schematic diagram of a scenario to which a voice interaction method provided in an embodiment of the present application may be applicable. Figure 8 Interface 801 in the figure shows a schematic diagram of a chat interface of an instant messaging APP for a mobile phone. In interface 801, when user A is chatting with user B, the electronic device can provide the user with a selection entry for voice input method 1, for example, button 1 associates voice input 1 with voice input mode 1. The user can click button 1 to select voice input method 1, and the voice input mode 1 corresponding to voice input method 1 can be, for example, a direct input mode based on voice. In voice input mode 1, the user can input voice 1 to the electronic device. Accordingly, the electronic device can recognize voice 1 and input text 1 obtained by recognizing the voice 1 in the input box of the chat interface.

[0140] Taking the English language scenario as an example, when a user expects to express the speech corresponding to the following text in speech 1, "Please buy me some flours". Because homophones or near-phonetic words are involved, the user's intended expression "flour" is mistakenly recognized as "flower", so that the text 1 displayed in the input box is: "Please buy me some flowers". Therefore, the wrong text needs to be modified.

[0141] In interface 802, the electronic device may also provide the user with a button 2, which is associated with a voice input method 2 and a voice input mode 2. For example, the voice input mode 2 may be a text editing mode based on voice. The user may click button 2 in interface 802 to switch to voice input method 2. In voice input mode 2, the user may input voice 2 to the electronic device to input a supplementary explanation or description of voice 1 to the electronic device. Among them, for homophone characters to be replaced, voice 2 may describe the homophone characters in at least one of the following ways: a common character combination description method; a character decomposition description method; a semantic description method. For example, in interface 803, voice 2 may include the voice corresponding to the following text: "flower change to flour which could be used to make bread", which means that "flower" in text 1 is changed to: the word "flour" for flour that can be used to make bread. Here, "the word for flour that can be used to make bread" is a semantic description method for "flour".

[0142] The electronic device can perform speech recognition on the voice 2 input by the user, and display the text 2 corresponding to the voice 2 in the echo area next to the button 2, as shown in interface 803. At the same time, the electronic device can obtain the semantic analysis result of the voice 2, and perform corresponding operations according to the semantic analysis result. For example, the erroneous text "flower" in interface 802 and interface 803 is modified to "flour", as shown in interface 804.

[0143] It should be noted that Figure 8 The English language scenario and chat interface are used as examples for illustration only and do not constitute a limitation on the language scenario. In other language scenarios, such as Chinese, German, French, etc., the same Figure 8 In a similar manner, in different voice input modes, voice information is input to the electronic device. The electronic device can perform corresponding operations according to the recognition result and / or semantic analysis result of the voice information, which will not be described in detail here.

[0144] Voice interaction scenario F:

[0145] In a voice interaction scenario, the electronic device can provide the user with a variety of voice input method selection entries on the display screen for the user to choose from, and these multiple voice input methods correspond one to one to multiple voice input modes. If the user's selection operation on a certain voice input method is detected, the electronic device can perform corresponding operations according to the voice information input by the user in the corresponding voice input mode. These operations can be cross-application operations. For example, the current interface on the display screen is the interface of application 1. When the user's authorization is obtained, a cross-application operation can be: the user obtains the information of application 2 through voice input in the current interface. Among them, application 2 may belong to the same developer as the electronic device, or application 2 may belong to a different developer from the electronic device, and this embodiment of the present application does not limit this.

[0146] See also Fig. 9 FIG. 1 is a schematic diagram of a scenario in which a voice interaction method provided in an embodiment of the present application may be applicable. Fig. 9 Interface 901 in the figure shows a chat diagram of an instant messaging APP of a mobile phone. In interface 901, the electronic device can provide the user with a selection entry for voice input method 2, which is represented by button 2. At this time, button 2 is in an inactive state and is drawn with a thinner line, indicating that the voice input method 2 associated with button 2 is inactive. The user can click button 2 in interface 901 to switch button 2 to an active state, as shown in interface 902, where button 2 in an active state is drawn with a thicker line.

[0147] In interface 902, in voice input mode 2 corresponding to voice input method 2, for example, a mode for obtaining cross-application target information based on voice, a user can input voice 2 to an electronic device. Voice 2 is used to express that the user expects to obtain target information from application 2. As an example, as shown in interface 902, the voice 2 may include a voice corresponding to the following text: "Insert my ID number (in the input box)". The electronic device can recognize voice 2 and display the voice recognition result corresponding to voice 2, that is, text 2 corresponding to voice 2, in the echo area next to button 2. At the same time, the electronic device can also obtain the semantic analysis result of voice 2, and know that the ID number of user A needs to be inserted in the input box of the chat interface. The electronic device can perform a cross-application target information acquisition operation. For example, application 2 is "Xiaoyi Memory", and the target information includes the ID number of the owner (user A). Accordingly, the electronic device can obtain the ID number of user A from "Xiaoyi Memory" according to the semantic analysis result of voice 2, expressed as 1234567890. The electronic device can insert the acquired target information into the input box of the chat interface, as shown in text 3 in interface 902.

[0148] Similarly, when user A wants to insert other target information obtained across applications into the input box of the chat interface, such as user A's contact address and other personal information, or weather information of a country or region, or multimedia resources such as music, audio and video, or information related to user A's food, clothing, housing, transportation and other services, such as hotel addresses and phone numbers, takeaway phone numbers, etc., user A can Fig. 9 In the manner shown, by selecting a corresponding voice input method, voice interaction is performed with the electronic device in the voice input mode corresponding to the voice input method, thereby freeing the user's hands and improving the accuracy and convenience of voice input.

[0149] Optionally, when user A expects (or receives information from user B) to perform a cross-application target operation, such as booking a hotel or ordering takeout, user A can Fig. 9 In the manner shown, by selecting a corresponding voice input method, in the voice input mode corresponding to the voice input method, for example, in a mode of cross-application target operation based on voice indication, voice interaction is performed with the electronic device, and voice indication information is input into the electronic device to call the target application (such as travel APP, food delivery APP, etc.) to perform the target operation or to make a phone call to book a hotel or food delivery, etc., thereby freeing the user's hands and improving the accuracy and convenience of voice input.

[0150] Voice interaction scenario G:

[0151] See also Fig.10 FIG. 1 is a schematic diagram of a scenario in which a voice interaction method provided in an embodiment of the present application may be applicable. Fig.10The interface 1001 in the figure shows a chat diagram of an instant messaging APP of a mobile phone. In the interface 1001, the electronic device can provide the user with a selection entry of the voice input method 2, which is represented by button 2. At this time, button 2 is in an inactive state and is drawn with a thinner line, indicating that the voice input method 2 associated with button 2 is inactive. A keyboard can also be displayed in the interface 1001. The user can click button 2 in the interface 1001 to switch button 2 to an active state, as shown in the interface 1002, where button 2 in an active state is drawn with a thicker line.

[0152] In interface 1002, in voice input mode 2 corresponding to voice input method 2, such as a voice-based help writing input mode, the user can input voice 2 to the electronic device, and voice 2 is used to express the user's desire to switch input methods. As an example, as shown in interface 1002, the voice 2 may include a voice corresponding to the following text: "Switch to Xiaoyi to help write". The electronic device can recognize voice 2 and display the voice recognition result corresponding to voice 2, that is, text 2 corresponding to voice 2, in the echo area next to button 2. At the same time, the electronic device can also obtain the semantic analysis result of voice 2, and know that it is necessary to switch the input method to "Xiaoyi to help write" in the chat interface. The electronic device can perform the operation of switching the input method, so that the keyboard displayed on interface 1002 is switched to the Xiaoyi help writing mode shown in interface 1003, so that in the Xiaoyi help writing mode, the electronic device can send greetings, blessings, etc. to user B instead of user A.

[0153] Similarly, if user A wishes to switch the voice input mode to at least one of the voice-based translation input mode, the voice-based shooting input mode, the voice-based traditional Chinese input mode, and the voice-based memory input mode, user A can follow Fig.10 The similar method shown in the figure performs voice interaction with the electronic device to switch the voice input mode to at least one of a voice-based translation input mode, a voice-based shooting input mode, a voice-based traditional Chinese input mode, a voice-based writing input mode, and a voice-based memory input mode. The implementation details can be found in the above combined Fig.10 The relevant description will not be repeated here.

[0154] It should be noted that Figure 5-Figure 10 Although some voice input modes are illustrated using the same button icon, in actual applications, one voice input mode may correspond to one button, or one button may be associated with several voice input modes, and this embodiment of the present application does not limit this.

[0155] It should be noted that Figure 5-Figure 10In other embodiments, the selection entry of different voice input methods provided by the electronic device may have other implementation methods. According to the user's selection of different voice input methods, the operation performed by the electronic device is not limited to Figure 5-Figure 10 The operations described.

[0156] Taking the example that the selection entrances of different voice input methods provided by the electronic device may have other implementation methods, the following takes the electronic device as a mobile phone as an example to introduce several possible implementation methods of the selection entrance of the voice input method in the embodiment of the present application.

[0157] In the embodiment of the present application, the electronic device may provide different buttons to associate with different voice input methods, one button is associated with one voice input method, and one voice input method corresponds to one voice input mode.

[0158] In the embodiment of the present application, any voice input method-associated voice input method key provided by the electronic device may be a physical key (such as a physical key) or a virtual key.

[0159] In some optional implementations, different presentation forms of the same key may be used to represent different keys, and a voice input method is associated with a presentation form and corresponds to a voice input mode. The embodiment of the present application does not limit the implementation method of different keys associated with the voice input function. In optional implementations, different states may also be set for the same key to indicate whether the function or mode associated with the key is activated. For example, the state of a key may include an activated state and an unactivated state. The activated state may also be referred to as a function activation state, indicating that the function / mode associated with the key is activated or activated. The unactivated state may also be referred to as an unactivated state, indicating that the function / mode associated with the key is not activated or activated.

[0160] In a possible implementation, taking the case where the keys associated with multiple voice input methods are all implemented as virtual keys, the user's selection operation on the selection entrances of the multiple voice input methods may include a click operation and / or a long press operation. Taking the case where the keys associated with multiple voice input methods are implemented as physical keys, the user's selection operation on the selection entrances of the multiple voice input methods may include a press operation and / or a long press operation.

[0161] The following descriptions are all made by taking a floating button as an example.

[0162] Example 1:

[0163] like Fig.11As shown, initially, in interface 1101, the electronic device can provide the user with a selection entry for a voice input method (such as voice input method 1), represented by button 1, and button 1 is in an inactive state. The user can perform one or more selection operations along the direction of the solid arrow. For example, the user can click button 1 in interface 1101 to select voice input method 1. Accordingly, the electronic device can detect the user operation and control button 1 from an inactive state to an active state according to the user operation, as shown in interface 1102. In interface 1102, in voice mode 1 corresponding to voice input method 1, the user can input voice 1 to the electronic device. The user can long press button 1 in interface 1101, or long press button 1 in interface 1102 to enter interface 1103. A function menu can be displayed in interface 1103. The function menu may include selection entries for multiple input methods, such as button 1, button 2, keyboard buttons, and delete buttons. Among them, button 1 is associated with Figure 5-Figure 10 The voice input method 1 introduced in the introduction corresponds to voice input mode 1. Button 2 is associated with Figure 5-Figure 10 The voice input method 2 introduced in the above corresponds to voice input mode 2. The user can click button 2 in interface 1103 to enter interface 1104. At this time, button 2 is in an activated state, and the voice input method 2 associated with button 2 is activated, and the user can input voice information to the electronic device in voice input mode 2. The user can also long press button 2 in interface 1104 to enter the function menu in interface 1103, so that the user can select other input methods from the menu.

[0164] Example 2:

[0165] In an optional implementation, initially, in interface 1103, the electronic device may provide the user with a function menu, including multiple input method selection entries, such as key 1, key 2, keyboard keys, and delete keys. The user may perform one or more selection operations along the direction of the dotted arrow. For example, the user may click key 1 in interface 1103, select voice input method 1, and enter interface 1103. Alternatively, the user may click key 2 in interface 1103, select voice input method 2, and enter interface 1104. The user may also click key 1 in interface 1102 to cancel voice input method 1 and enter interface 1101, where key 1 is in an inactive state.

[0166] It should be understood that Fig.11Lines of different thicknesses are used to indicate different states of a key, for example, a thin line indicates that the input method associated with the corresponding key is in an inactive state, and a thick line indicates that the input method associated with the corresponding key is in an active state. In other embodiments, different key states can also be distinguished by whether the key is highlighted, for example, a highlighted state indicates an activated state, and a non-highlighted state indicates an inactive state. The embodiment of the present application does not limit the presentation form of the key state.

[0167] Example 3:

[0168] In another possible implementation, Fig.12 As shown, initially, in interface 1201, the electronic device can provide the user with a function menu, including multiple voice input method selection entrances, such as key 1 and key 2, key 1 is associated with Figure 5-Figure 10 The voice input method 1 introduced in the above corresponds to voice input mode 1. Key 2 is associated with Figure 5-Figure 10 The voice input method 1 introduced in the above corresponds to the voice input mode 2. In the interface 1201, both the key 1 and the key 2 are in an inactive state. For easy distinction, the inactive keys 1 and 2 are drawn with thinner lines.

[0169] In interface 1201, the user can select voice input method 2 associated with button 2 by clicking button 2, and enter interface 1202. At this time, button 2 is in an activated state. In voice input mode 2, the user can input voice to the electronic device. In interface 1201, the user can select voice input method 1 associated with button 1 by clicking button 1, and enter interface 1203. At this time, button 1 is in an activated state. In voice input mode 1, the user can input voice to the electronic device.

[0170] Optionally, the user can return to the menu shown in interface 1201 by long pressing button 2 in interface 1202, or long pressing button 1 in interface 1203. At this time, all buttons in the menu are in an inactive state.

[0171] Understandably, Fig.11 Or the menu in 12 is only an example and not a limitation. In other embodiments, the menu may be presented in other forms, and the buttons in the menu may include other buttons besides the buttons associated with the various voice input methods.

[0172] Example 4:

[0173] See also Fig.13 As shown in the interface 1301, the electronic device can provide the user with a function menu in the form of a disc, and the menu includes multiple input method selection entries, such as key 1, key 2, keyboard and delete key. Among them, key 1 is associated with Figure 5-Figure 10The voice input method 1 introduced in the introduction corresponds to voice input mode 1. Button 2 is associated with Figure 5-Figure 10 The voice input method 2 introduced in corresponds to the voice input mode 2. In the interface 1301, each key in the menu is in an inactive state. For easy distinction, each key in an inactive state is drawn with a thinner line.

[0174] In interface 1301, the user can select voice input method 2 associated with button 2 by clicking button 2, and enter interface 1302. At this time, button 2 is in an activated state. In voice input mode 2, the user can input voice to the electronic device. In interface 1301, the user can select voice input method 1 associated with button 1 by clicking button 1, and enter interface 1303. At this time, button 1 is in an activated state. In voice input mode 1, the user can input voice to the electronic device.

[0175] Optionally, the user can return to the menu shown in interface 1301 by long pressing button 2 in interface 1302, or long pressing button 1 in interface 1303. At this time, all buttons in the menu are in an inactive state.

[0176] akin, Fig.12 and Fig.13 In the example, lines of different thicknesses indicate that the button is in different states, for example, a thin line indicates that the corresponding button is in an inactive state, and a thick line indicates that the corresponding button is in an activated state. In other embodiments, different button states can also be distinguished by whether the button is highlighted, and the embodiment of the present application does not limit the presentation form of the button state.

[0177] It should be understood that Figure 11-13 The above is only a schematic representation of the fact that buttons associated with multiple voice input methods can be displayed on the display screen of the electronic device for the user to select, and does not limit the specific type or specific content of the current interface on the display screen. Figure 11-13 The buttons shown can be displayed in a suspended state on the mobile phone desktop, or in a suspended state on an interface corresponding to various possible business scenarios, such as Figure 5 The memo interface shown, or for example Figure 6a or Figure 6b The display interface of the reading APP shown, or for example Figure 7 The operation interface of the phone APP shown, or for example Figure 8-Figure 10 The chat interface of the instant messaging APP shown is not limited to this embodiment of the present application.

[0178] Based on the different implementations described above, the application scenarios of the voice interaction method of the embodiment of the present application and the implementation methods of the selection entrances of multiple voice input methods provided by the electronic device are introduced. Figure 5-Figure 10 The voice interaction scenario shown, Figure 11-13 The different implementation methods of the selection entrance shown are used to introduce the implementation process of the voice interaction method of the embodiment of the present application, so as to understand the interface display effect that can be achieved by using the voice interaction method of the embodiment of the present application.

[0179] Scenario 1:

[0180] See also Fig.14 As shown, the voice interaction method may include the following steps:

[0181] S1401: The electronic device provides a selection entry for multiple voice input methods on the display screen.

[0182] In the embodiment of the present application, the multiple voice input methods correspond one-to-one to the multiple voice input modes provided by the electronic device.

[0183] For example, the previous text Figure 5-Figure 13 In the introduction, button 1 is the selection entry for voice input method 1, and voice input method 1 corresponds to voice input mode 1. Button 2 is the selection entry for voice input method 2, and voice input method 2 corresponds to voice input mode 2. The multiple voice input modes may include at least one of the following: voice-based direct input mode; voice-based text editing mode; voice-based text query mode; voice-based text creation mode; voice-based translation input mode; voice-based shooting input mode; voice-based traditional Chinese input mode; voice-based writing input mode; voice-based memory input mode; voice-based mode for obtaining cross-application target information; voice-based mode for indicating cross-application target operations. Refer to the above in conjunction with Figure 5-Figure 10 The various voice interaction scenarios introduced above will not be repeated here.

[0184] S1402: User A performs a selection operation on multiple voice input methods. Accordingly, the electronic device detects the user's selection operation on multiple voice input methods.

[0185] In the embodiment of the present application, multiple voice input methods can correspond to multiple keys one by one, and the multiple keys can be physical keys or virtual keys. As an example, the multiple keys can be the above combined Figure 5-Figure 13 The different virtual keys described above, the user's selection operation for multiple voice input methods may include a click operation and / or a long press operation. Alternatively, the multiple keys may be different physical keys, and the user's selection operation for multiple voice input methods may include a press operation and / or a long press operation.

[0186] In one example, if multiple voice input method selection entrances are directly provided on the display screen in S1401, such as multiple buttons, the user can perform a selection operation by clicking a button to select a first voice input method from the multiple voice input methods, such as Fig.11 As shown in the interface 1103, or as Fig.12 As shown in the interface 1201, or as Fig.13 As shown in interface 1301.

[0187] In another example, if in S1401, multiple voice input method selection entrances are not directly provided on the display screen, the user can perform a second operation on a first button provided on the display screen, where the first button is any one of the multiple buttons. After the electronic device detects the user's second operation on the first button, it can provide multiple voice input method selection entrances on the display screen according to the second operation. As an example, the second operation may include Fig.11 A long press operation on button 1 in interface 1101 or a long press operation on button 2 in interface 1102 can call out a function menu in interface 1103, which is a selection entrance for multiple voice input methods.

[0188] It should be understood that in S1402, the selection operation performed by the user on the selection entry of the multiple voice input methods may also be other types of operations. For example, if the selection entry is implemented as a physical button, the selection operation may be a pressing operation or a long pressing operation on the corresponding physical button, which will not be described in detail here.

[0189] S1403: The electronic device determines a first input method from multiple voice input methods according to the selection operation. The first voice input method corresponds to a first voice input mode among multiple voice input modes. Accordingly, user A performs voice interaction with the electronic device in the first voice input mode.

[0190] For example Fig.11 In the interface 1103, the electronic device detects a user's click operation on button 1, selects voice input method 1 as the first input method, enters interface 1102, and performs voice interaction with the user in voice input mode 1. The voice input mode 1 is the first voice input mode. Alternatively, in the interface 1103, the electronic device detects a user's click operation on button 2, selects voice input method 2 as the first input method, enters interface 1104, and performs voice interaction with the user in voice input mode 2, and the voice input mode 2 is the first voice input mode.

[0191] or for example Fig.12In the interface 1201, the electronic device detects a user's click operation on button 2, selects voice input 2 as the first voice input method, enters interface 1202, and performs voice interaction with the user in voice input mode 2, which is the first voice input mode. Alternatively, in the interface 1201, the electronic device detects a user's click operation on button 1, selects voice input method 1 as the first input method, enters interface 1203, and performs voice interaction with the user in voice input mode 1, which is the first voice input mode.

[0192] or for example Fig.13 In interface 1301, the electronic device detects a user's click operation on button 2, selects voice input 2 as the first voice input method, enters interface 1302, and performs voice interaction with the user in voice input mode 2, which is the first voice input mode. Alternatively, in interface 1301, the electronic device detects a user's click operation on button 1, selects voice input method 1 as the first input method, enters interface 1303, and performs voice interaction with the user in voice input mode 1, which is the first voice input mode.

[0193] S1404: The electronic device performs a first operation in a first voice input mode according to first voice information input by a user.

[0194] In the embodiment of the present application, in different voice interaction scenarios, the first voice information and the first operation may be different.

[0195] For example, if the first voice input mode is a voice-based text editing mode and is a replacement for homophones or similar-sounding characters, the first voice information may describe the first character in at least one of the following ways: a common character combination description method, a character decomposition description method, or a semantic description method. In S1404, the electronic device may replace the second character displayed on the display screen with the first character, wherein the first character and the second character are homophones or similar-sounding characters. Figure 7 or Figure 8 The content introduced.

[0196] Or for example, if the first voice input mode is a voice-based text editing mode, and it is a replacement of non-homophone characters or similar-sounding characters, the first voice information can indicate the old text content to be modified, the new text content to be modified, and a keyword, and the keyword is associated with an editing operation. For example, "change to", "change to", "modify to", "modify to", "change to", "replace with", etc. can all be keywords associated with modification operations. In S1404, the electronic device can modify the old text content to the new text content. See the above in conjunction with Figure 5The content introduced.

[0197] Or for example, if the first voice input mode is a voice-based text editing mode, the first voice information can be used to describe a new first text format. In S1404, the electronic device can modify the second text format to the first text format according to the first voice information. Figure 5 In the content introduced, text 1 adopts the second text format, text 3 adopts the first text format, and the editing operation is a text wrap operation.

[0198] Or for example, if the first voice input mode is a voice-based text query mode, the first voice information can be used to describe the query instruction. In S1404, the electronic device can perform a query operation in the first text information according to the first voice information. Figure 6a The content introduced is to search for the number of words in the text shown in interface 602, or to search for XX in the text shown in interface 603.

[0199] Or for example, if the first voice input mode is a voice-based text creation mode, the first voice information can be used to indicate switching to an input method associated with the voice-based text creation mode, such as Figure 6b Alternatively, the first voice information may be used to describe the requirements to be met by the creation, such as birthday wishes, holiday wishes, etc. In S1404, the electronic device may perform a text creation operation according to the first voice information.

[0200] Or for example, if the first voice input mode is a mode for acquiring cross-application target information based on voice, the first voice information can be used to describe the target information. Fig. 9 As shown in FIG. 1 , “My ID Number” is a description of user A’s personal information. In S1404, the electronic device can receive the first voice information input by the user in the first interface provided by the electronic device in the first voice input mode. Then, the electronic device can obtain the target information from the second application according to the first voice information input by the user, and display the target information on the first interface. Fig. 9 As shown, user A's ID number is entered and displayed in the input box of the chat interface.

[0201] Scenario 2:

[0202] In an optional implementation, the voice interaction method of the embodiment of the present application can be implemented by the electronic device and the server in collaboration. Figure 4 In the architecture shown in A, the electronic device interacts with the server to obtain Figure 5-Figure 10 The interface display effect achieved by the voice interaction scenario shown.

[0203] See also Fig.15As shown, the voice interaction method may include the following steps:

[0204] S1501: The electronic device receives voice information 1 (an example of the first voice information) input by the user in voice input mode 1 (an example of the first voice input mode). The implementation details of this step can be found in the above combined Figure 5-Figure 14 The relevant description will not be repeated here.

[0205] S1502: The electronic device sends a request message 1 (an example of a first request message) to the server 1, where the request message 1 includes voice information 1. Accordingly, the server 1 receives the request message 1 and obtains the voice information 1 from the request message 1.

[0206] The server 1 may be a server deployed with a large-scale language model (referred to as a large model), and any application on the electronic device may be authorized to communicate with the server 1 .

[0207] S1503: Server 1 performs semantic analysis on voice information 1 to obtain a semantic analysis result.

[0208] S1504: Server 1 sends response information 1 (an example of first response information) to the electronic device, where the response information 1 includes a semantic analysis result. Accordingly, the electronic device receives the response information 1.

[0209] In an optional implementation, in S1502, the electronic device may send a request message 1 to the server 1 through the server 2. The server 1 is an application server, and may parse and repackage the request message 1 from the electronic device and send it to the server 1. Accordingly, in S1504, the server 2 may parse and repackage the response message 1 from the server 1 and send it to the electronic device.

[0210] S1505: The electronic device performs a first operation according to the semantic analysis result. Figure 5-Figure 13 The relevant description will not be repeated here.

[0211] Scenario 3:

[0212] In an optional implementation, the voice interaction method of the embodiment of the present application can be implemented by the electronic device and the server in collaboration. Figure 4 In the architecture shown in A, the electronic device interacts with the server to obtain Figure 5-Figure 10 The interface display effect achieved by the voice interaction scenario shown.

[0213] See also Fig.16 As shown, the voice interaction method may include the following steps:

[0214] S1601: The electronic device receives voice information 1 (an example of the first voice information) input by the user in voice input mode 1 (an example of the first voice input mode). The implementation details of this step can be found in the above combined Figure 5-Figure 15 The relevant description will not be repeated here.

[0215] S1602: The electronic device sends a request message 1 (an example of a second request message) to the server 3, where the request message 1 includes voice information 1. Accordingly, the server 3 receives the request message 1 and obtains the voice information 1 from the request message 1.

[0216] The server 3 may be a server equipped with an ASR module, and any application on the electronic device may be authorized to communicate with the server 3 .

[0217] S1603: Server 3 performs voice recognition on voice information 1 to obtain text information 1 (an example of the first text information).

[0218] S1604: the server 3 sends a response message 1 (an example of the second response message) to the electronic device, where the response message 1 includes text information 1. Accordingly, the electronic device receives the response message 1 from the server 3.

[0219] S1605: The electronic device displays text information 1 on the display screen. Figure 5-Figure 13 The relevant description will not be repeated here.

[0220] Scenario 4:

[0221] In this scenario, the electronic device can interact with different servers to respectively implement speech recognition and / or semantic analysis of the voice information to obtain Figure 5-Figure 10 The interface display effect achieved by the voice interaction scenario shown.

[0222] like Fig.17 As shown, the method comprises the following steps:

[0223] S1701: The electronic device receives voice information 1 input by the user in voice input mode 1 (an example of the first voice input mode). The implementation details of this step can be found in the above combined Figure 5-Figure 15 The relevant description will not be repeated here.

[0224] S1702: The electronic device sends a request message 1 (an example of a second request message) to the server 3, wherein the request message 1 includes voice information 1 (an example of a first voice information). Accordingly, the server 3 receives the request message 1 and obtains the voice information 1 from the request message 1.

[0225] The server 3 may be a server equipped with an ASR module, and any application on the electronic device may be authorized to communicate with the server 3 .

[0226] S1703: Server 3 performs voice recognition on voice information 1 to obtain text information 1 (an example of the first text information).

[0227] S1704: the server 3 sends a response message 1 (an example of the second response message) to the electronic device, where the response message 1 includes text information 1. Accordingly, the electronic device receives the response message 1 from the server 3.

[0228] S1705: The electronic device displays text information 1 on the display screen. Figure 5-Figure 13 The relevant description will not be repeated here.

[0229] S1706: The electronic device sends a request message 2 (an example of a first request message) to the server 1, where the request message 2 includes the voice information 1. Accordingly, the server 1 receives the request message 2 and obtains the voice information 1 from the request message 2.

[0230] The server 1 may be a server deployed with a large-scale language model (referred to as a large model), and any application on the electronic device may be authorized to communicate with the server 1 .

[0231] S1707: Server 1 performs semantic analysis on voice information 1 to obtain a semantic analysis result.

[0232] S1708: Server 1 sends response information 1 (an example of first response information) to the electronic device, where the response information 1 includes a semantic analysis result. Accordingly, the electronic device receives the response information 1.

[0233] In an optional implementation, in S1706, the electronic device may send a request message 2 to the server 1 through the server 2. The server 2 is an application server, and may parse and repackage the request message 2 from the electronic device and send it to the server 1. Accordingly, in S1708, the server 2 may parse and repackage the response message 2 from the server 1 and send it to the electronic device.

[0234] S1709: The electronic device performs a first operation according to the semantic analysis result. Figure 5-Figure 13 The relevant description will not be repeated here.

[0235] It should be understood that in the embodiment of the present application, S1702-S1704 and S1706-S1708 are not limited to the execution sequence, and the electronic device can execute S1702-S1704 and S1706-S1708 in parallel, or execute S1702-S1704 first and then execute S1706-S1708.

[0236] Scenario 5:

[0237] In this scenario, the electronic device can implement the voice interaction method of the embodiment of the present application by editing the text content with recognition errors through voice input, such as Figure 5 or Figure 7 or Figure 8 The voice interaction scenario shown. Among them, in different voice interaction stages, the user can use different voice input modes to input voice information to the electronic device, and these voice input modes are all examples of the first voice input mode. The voice information input by the user in the corresponding voice input mode is an example of the first voice information. In the interaction between the electronic device and different servers, request message 1 and request message 2 are different examples of the second request message, and request message 3 is an example of the first request message. Correspondingly, response message 1 and response message 2 are different examples of the second response message, and response message 3 is an example of the first response message. Text message 1 and text message 2 are different examples of the first text message.

[0238] like Fig.18 As shown, the voice interaction method may include the following steps:

[0239] S1801: The electronic device receives voice information 1 input by the user in voice input mode 1. The implementation process of S1801 can be found in Fig.14 The above is combined with the introduction of S1401-S1404, which will not be repeated here.

[0240] S1802: The electronic device sends a request message 1 to the server 3, wherein the request message 1 includes the voice information 1. Accordingly, the server 3 receives the request message 1 and obtains the voice information 1 from the request message 1.

[0241] The server 3 may be a server equipped with an ASR module, and any application on the electronic device may be authorized to communicate with the server 3 .

[0242] S1803: Server 3 performs voice recognition on voice information 1 to obtain text information 1.

[0243] S1804: The server 3 sends a response message 1 to the electronic device, where the response message 1 includes text message 1. Accordingly, the electronic device receives the response message 1 from the server 3.

[0244] S1805: The electronic device displays text information 1 on the display screen.

[0245] S1806: The electronic device receives the voice information 2 input by the user in the voice input mode 2. The implementation details of this step can be found in the above combined Figure 5-Figure 15 The relevant description will not be repeated here.

[0246] S1807: The electronic device sends a request message 2 to the server 3, wherein the request message 2 includes the voice information 2. Accordingly, the server 3 receives the request message 2 and obtains the voice information 2 from the request message 2.

[0247] S1808: Server 3 performs voice recognition on voice information 2 to obtain text information 2.

[0248] S1809: The server 3 sends a response message 2 to the electronic device, where the response message 2 includes text message 2. Accordingly, the electronic device receives the response message 2 from the server 3.

[0249] S1810: The electronic device displays text information 2 on the display screen.

[0250] S1811: The electronic device sends a request message 3 to the server 1, wherein the request message 3 includes the voice information 2. Accordingly, the server 1 receives the request message 3 and obtains the voice information 2 from the request message 3.

[0251] The server 1 may be a server deployed with a large-scale language model (referred to as a large model), and any application on the electronic device may be authorized to communicate with the server 1 .

[0252] S1812: Server 1 performs semantic analysis on voice information 2 to obtain a semantic analysis result.

[0253] S1813: Server 1 sends response information 3 to the electronic device, where the response information 3 includes the semantic analysis result. Correspondingly, the electronic device receives the response information 3.

[0254] In an optional implementation, in S1811, the electronic device may send a request message 3 to the server 1 through the server 2. The server 2 is an application server, and may parse and repackage the request message 3 from the electronic device and send it to the server 1. Accordingly, in S1813, the server 2 may parse and repackage the response message 3 from the server 1 and send it to the electronic device.

[0255] S1814: The electronic device performs a first operation according to the semantic analysis result. Figure 5-Figure 13 The relevant description will not be repeated here.

[0256] It should be understood that in the embodiments of the present application, S1802-S1804, S1807-S1809 and S1811-S1813 are not limited to the execution sequence, and the electronic device may execute S1802-S1804, S1807-S1809 and S1811-S1813 in parallel, or may first execute S1802-S1804, then execute S1807-S1809, and then execute S1811-S1813.

[0257] Based on the above embodiments, the present application further provides an electronic device, the electronic device comprising multiple functional modules; the multiple functional modules interact with each other to implement the functions performed by the electronic device in each method described in the embodiments of the present application. The multiple functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the multiple functional modules can be arbitrarily combined or divided based on the specific implementation. Fig.14 In the embodiment shown, the electronic device executes S1401, S1403-S1404; Fig.15 In the embodiment shown, the electronic device performs S1502 to step 1505, and also performs Fig.15 S1602 - S1605 are executed by the electronic device in the illustrated embodiment. Figure 17-Figure 18 Similar to .

[0258] Based on the above embodiments, the present application further provides an electronic device, which includes at least one processor and at least one memory, wherein the at least one memory stores computer program instructions, and when the electronic device is running, the at least one processor executes the functions executed by the electronic device in each method described in the embodiments of the present application. Fig.14 In the embodiment shown, the electronic device executes S1401, S1403-S1404; Fig.15 In the embodiment shown, the electronic device performs S1502 to step 1505, and also performs Fig.15 S1602 - S1605 are executed by the electronic device in the illustrated embodiment. Figure 17-Figure 18 Similar to .

[0259] Based on the above embodiments, the present application also provides a voice interaction system, which may include an electronic device and a server as in the previous embodiments.

[0260] Based on the above embodiments, the present application also provides a computer program product, which includes: a computer program (also referred to as code, or instructions), which enables a computer to execute the methods described in the embodiments of the present application when the computer program is executed.

[0261] Based on the above embodiments, the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a computer, the computer executes the methods described in the embodiments of the present application.

[0262] Based on the above embodiments, the present application further provides a chip, which is used to read a computer program stored in a memory to implement the methods described in the embodiments of the present application.

[0263] Based on the above embodiments, the present application provides a chip system, which includes a processor for supporting a computer device to implement the methods described in the embodiments of the present application. In a possible design, the chip system also includes a memory, which is used to store the necessary programs and data of the computer device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. It should be understood by those skilled in the art that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0264] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0265] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0266] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0267] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of protection of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A voice interaction method, characterized in that: Applied to an electronic device, the electronic device supports multiple voice input modes, the method comprises: Providing a selection entry for multiple voice input methods, wherein the multiple voice input methods correspond one to one with the multiple voice input modes; Detecting a user's selection operation of the plurality of voice input methods; Determine a first voice input method from the multiple voice input methods according to the selection operation, wherein the first voice input method corresponds to a first voice input mode among the multiple voice input modes; In the first voice input mode, a first operation is performed according to the first voice input by the user.

2. The method according to claim 1, characterized in that The multiple voice input modes include at least one of the following: Direct voice-based input mode; Voice-based text editing mode; Voice-based text query mode; Voice-based text creation mode; Voice-based translation input mode; Voice-based shooting input mode; Voice-based traditional Chinese input mode; Voice-based writing input mode; Voice-based memory input mode; A model for obtaining cross-application target information based on voice; Mode for voice-based instructions across application targets.

3. The method according to claim 2, characterized in that If the first voice input mode is a voice-based text editing mode, the first voice describes the first character in at least one of the following ways: a common character combination description way, a character decomposition description way, or a semantic description way; The performing a first operation according to the first voice input by the user includes: The displayed second character is replaced with the first character, and the first character and the second character are homophones or characters with similar pronunciations.

4. The method according to claim 2, characterized in that: If the first voice input mode is a mode for acquiring cross-application target information based on voice, performing a first operation according to the first voice input by the user in the first voice input mode includes: In the first voice input mode, receiving a first voice input by the user on a first interface provided by the electronic device; Acquiring target information from a second application according to the first voice input by the user; The target information is displayed on the first interface.

5. The method according to any one of claims 1 to 4, characterized in that The selection entry for providing multiple voice input methods includes: providing a first button; detecting a second operation of the user on the first button; According to the second operation, a selection entrance for the multiple voice input methods is provided, wherein the first button is any one of the multiple buttons, and the multiple buttons correspond one-to-one to the multiple voice input methods.

6. The method according to claim 5, characterized in that The first button is a physical button; or the first button is a virtual button.

7. The method according to any one of claims 1 to 6, characterized in that The multiple voice input methods correspond one-to-one to multiple keys, and the multiple keys are physical keys or virtual keys.

8. The method according to any one of claims 1 to 7, characterized in that The selection entrances of the multiple voice input methods are included in a function menu provided by the electronic device.

9. The method according to any one of claims 1 to 8, characterized in that The performing a first operation according to the first voice input by the user includes: Obtaining a semantic analysis result of the first speech; The first operation is performed according to the semantic analysis result.

10. The method according to claim 9, characterized in that The obtaining of the semantic analysis result of the first speech includes: Sending a first request message to a server, wherein the first request message includes the first voice; A first response message is received from the server, wherein the first response message includes a semantic analysis result of the first speech.

11. The method according to claim 10, characterized in that A large-scale language model is deployed on the server, and the large-scale language model is used to perform semantic analysis on the first speech to obtain the semantic analysis result.

12. The method according to any one of claims 1 to 11, characterized in that The performing a first operation according to the first voice input by the user further includes: Acquire first text information, where the first text information includes a speech recognition result of the first speech; The first text information is displayed.

13. The method according to claim 12, characterized in that The obtaining of the first text information includes: Sending a second request message to a server, where the second request message includes the first voice; A second response message is received from the server, where the second response message includes the first text message obtained by recognizing the first voice.

14. An electronic device, characterized in that: The system comprises at least one processor, wherein the at least one processor is coupled to at least one memory, and the at least one processor is configured to read a program stored in the at least one memory to execute the method according to any one of claims 1 to 13.

15. A readable storage medium, characterized in that: The readable storage medium stores instructions, which, when executed on a device, enable the device to execute the method according to any one of claims 1 to 13.

Images