A voice interaction processing method and related apparatus

Abstract

The application discloses a voice interaction processing method and related device, relates to natural language processing technology in the field of artificial intelligence, in particular processing technology of multi-turn dialogue, and the method comprises the following steps: an electronic device receives a first voice signal; in the case that the electronic device is connected with a cloud server, the electronic device uploads the first voice signal to the cloud server; when the cloud server processes voice data, the cloud server sends corresponding instructions to the electronic device, instructs the electronic device to perform corresponding actions, and synchronously sends the context (intention and slot information) of voice dialogue to the electronic device. In this way, if network interruption occurs in the multi-turn dialogue voice service, resulting in switching from processing on the cloud server to processing on the electronic device, the electronic device can also continue to perform the original voice service based on the context of the voice dialogue and the received next voice signal, thereby solving the problem of voice service interruption in the multi-turn dialogue.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a voice interaction processing method and related device. BACKGROUND

[0002] With the gradual development of voice interaction technology, more and more intelligent devices have voice interaction functions. Voice interaction refers to that a user inputs voice / text to obtain voice / text response, such as the user voice input "What is the weather today?", and the intelligent device voice returns "Sunny, 25-29 degrees".

[0003] The current voice interaction system needs network support. When some conditions (such as network interruption) occur, the original voice interaction service cannot continue to execute, affecting the user experience. SUMMARY

[0004] The embodiments of the present application provide a voice interaction processing method and related device to solve the problem of voice service interruption in multi-turn dialogue, and improve the voice service processing capability.

[0005] In a first aspect, the present application provides a voice interaction processing method, which is applied to an electronic device and includes: receiving, by the electronic device, an input first voice signal; uploading, by the electronic device, the first voice signal to a cloud server in a case where the electronic device establishes a connection with the cloud server; receiving, by the electronic device, first voice reply content, an intent, and one or more slot information corresponding to the intent, which are sent by the cloud server, the intent and the one or more slot information being obtained by the cloud server through identification of the first voice signal, and the first voice reply content being determined by the cloud server based on the intent and the one or more slot information; outputting, by the electronic device, the first voice reply content, receiving, by the electronic device, a second voice signal; identifying, by the electronic device, the second voice signal to obtain corresponding semantic information in a case where the communication quality between the electronic device and the cloud server is poor, determining, by the electronic device, a first operation based on the intent, the one or more slot information, and the semantic information; and executing, by the electronic device, the first operation.

[0006] In the process of processing voice services, the embodiments of the present application can be processed by the cloud server or the electronic device. When the cloud server processes voice data, the cloud server sends corresponding instructions to the electronic device to instruct the electronic device to perform corresponding actions, and synchronously sends the context (intent and slot information) of the voice dialogue to the electronic device. In this way, if network interruption occurs in the voice service of multi-turn dialogue, causing the voice service to switch from processing on the cloud server to processing on the electronic device, the electronic device can also continue to execute the original voice service based on the context of the voice dialogue and the received next voice signal, thereby solving the problem of voice service interruption in multi-turn dialogue and improving the voice service processing capability.

[0007] In a possible implementation, the electronic device determines the first operation based on the intent and the one or more slot information and the semantic information, including: the electronic device identifies that one of the semantic information and the one or more slot information is missing slot matching, fills the semantic information as the value of the slot; and the electronic device determines the first operation based on the intent and the filled one or more slot information. Here, the process of the electronic device processing the original voice service based on the second voice signal is specifically described. Since the electronic device obtains the intent and slot information corresponding to the first voice signal, the electronic device can continue to fill the slot based on the intent and slot information, and implement the capability of continuing to process the original voice service.

[0008] In a possible implementation, the first operation includes one or more of the following: playing the second voice reply content; displaying the text content of the second voice reply content; and jumping to a corresponding interface.

[0009] In a possible implementation, the method further includes: the electronic device receiving a first instruction sent by the cloud server; and the electronic device displaying the text content of the first voice reply content and / or jumping to a corresponding interface based on the first instruction.

[0010] In a possible implementation, the communication quality between the electronic device and the cloud server is poor, including: the electronic device fails to upload the second voice signal to the cloud server; or the electronic device uploads the first voice signal to the cloud server, and does not receive reply data from the cloud server within a preset time. Here, the time when the communication quality is poor is explained. The communication quality between the electronic device and the cloud server can be poor when the electronic device uploads the second voice signal, or the communication quality between the electronic device and the cloud server can be poor when the cloud server issues the voice reply content corresponding to the second voice signal.

[0011] In a possible implementation, the electronic device receives the first voice signal, including: the electronic device receives the first voice signal through a voice assistant application.

[0012] In a second aspect, the present application provides a voice interaction processing method, which is applied to a cloud server and includes: the cloud server receiving a first voice signal uploaded by an electronic device; the cloud server identifying the first voice signal to obtain a corresponding intent and one or more slot information corresponding to the intent, and determining first voice reply content based on the intent and the one or more slot information; and the cloud server sending the first voice reply content, the intent and the one or more slot information to the electronic device.

[0013] In the embodiment of the application, when the cloud server sends an instruction to the electronic device to instruct the electronic device to perform a corresponding action, the cloud server synchronously sends the context (intention and slot information) of the voice dialogue to the electronic device. In this way, if network interruption occurs in the multi-round dialogue voice service, causing the voice service to be processed from the cloud server to the electronic device, the electronic device can continue to perform the original voice service based on the context of the voice dialogue and the received next voice signal, thereby solving the problem of interruption of the multi-round dialogue voice service and improving the voice service processing capability.

[0014] In a possible implementation, the cloud server sends the first voice reply content, the intention and the one or more slot information to the electronic device, including: the cloud server sends the first voice reply content, the intention and the one or more slot information to the electronic device in the case that there is a missing in at least one of the one or more slot information. Here, a case is provided in which the cloud server sends the intention and the slot information to the electronic device, that is, when there is a missing in the slot information, it is determined that the current voice service is a multi-round dialogue service, and then the cloud server sends the intention and the slot information to the electronic device; if there is no missing in the slot information, the voice service can be completed in a single round and there is no need to obtain the next voice signal. Through the judgment step of whether the slot information is missing, further decision is made on the intention and the slot information, which can save resources.

[0015] In a third aspect, the application provides a voice interaction processing system, the voice interaction processing system comprising an electronic device and a cloud server, wherein

[0016] The electronic device is configured to receive a first voice signal.

[0017] The electronic device is further configured to upload the first voice signal to the cloud server in the case that the electronic device is connected to the cloud server.

[0018] The cloud server is configured to recognize the first voice signal to obtain corresponding intention and one or more slot information corresponding to the intention, and determine first voice reply content based on the intention and the one or more slot information.

[0019] The cloud server is further configured to send the first voice reply content, the intention and the one or more slot information to the electronic device.

[0020] The electronic device is further configured to receive a second voice signal after outputting the first voice reply content.

[0021] The electronic device is further configured to recognize the second voice signal to obtain corresponding semantic information in the case that the communication quality between the electronic device and the cloud server is poor, and determine a first operation based on the intention, the one or more slot information and the semantic information.

[0022] The electronic device is also configured to perform the first operation.

[0023] In the process of processing the voice service, the embodiments of the present application can be processed by the cloud server or the electronic device. When the cloud server processes the voice data, the cloud server sends corresponding instructions to the electronic device, instructs the electronic device to perform corresponding actions, and synchronously sends the context (intention and slot information) of the voice conversation to the electronic device. In this way, if the network interruption occurs in the multi-round conversation voice service, causing the voice service to switch from processing on the cloud server to processing on the electronic device, the electronic device can also continue to perform the original voice service based on the context of the voice conversation and the received next voice signal, thereby solving the problem of voice service interruption in multi-round conversation and improving the voice service processing capability.

[0024] In a possible implementation, the electronic device is further configured to, when it is identified that one of the semantic information and the one or more slot information is missing, fill the semantic information as the value of the slot; and determine the first operation based on the intention and the one or more filled slot information. Here, the process of the electronic device processing the original voice service based on the second voice signal is specifically described. Since the electronic device obtains the intention and slot information corresponding to the first voice signal, it can continue to fill the slot based on the intention and slot information, and realize the capability of continuing to process the original voice service.

[0025] In a possible implementation, the first operation includes one or more of the following: playing the second voice reply content; displaying the text content of the second voice reply content; and jumping to a corresponding interface.

[0026] In a possible implementation, the electronic device is further configured to receive the first instruction sent by the cloud server; and display the text content of the first voice reply content based on the first instruction, and / or jump to a corresponding interface.

[0027] In a possible implementation, the communication quality between the electronic device and the cloud server is poor, including: the electronic device fails to upload the second voice signal to the cloud server; or the electronic device uploads the first voice signal to the cloud server, and does not receive the reply data of the cloud server within a preset time. Here, the time when the communication quality is poor is explained. The communication quality between the electronic device and the cloud server can be poor when the electronic device uploads the second voice signal, or the communication quality between the electronic device and the cloud server can be poor when the cloud server issues the voice reply content corresponding to the second voice signal.

[0028] In a possible implementation, the electronic device is further configured to receive the first voice signal through a voice assistant application.

[0029] In a possible implementation, the cloud server is further configured to send the first voice reply content, the intent, and the one or more slot information to the electronic device in a case where the one or more slot information is incomplete. Here, a case is provided in which the cloud server sends the intent and the slot information to the electronic device, i.e., when the slot information is incomplete, it is determined that the current voice service is a multi-turn dialogue service, and then the cloud server sends the intent and the slot information to the electronic device; if the slot information is not incomplete, the voice service can be processed in a single turn, and there is no need to obtain the next voice signal. Through the judgment step of whether the slot information is incomplete, further determination of the intent and the slot information is performed, and resources can be saved.

[0030] In a fourth aspect, the present application provides an electronic device, comprising: one or more processors, one or more memories; the one or more memories are coupled to the one or more processors; the one or more memories are configured to store computer program codes, the computer program codes comprising computer instructions; when the computer instructions run on the processors, the electronic device executes the voice interaction processing method in any possible implementation of the first aspect.

[0031] In a fifth aspect, the present application provides a cloud server, comprising: one or more processors, one or more memories; the one or more memories are coupled to the one or more processors; the one or more memories are configured to store computer program codes, the computer program codes comprising computer instructions; when the computer instructions run on the processors, the electronic device executes the voice interaction processing method in any possible implementation of the second aspect.

[0032] In a sixth aspect, the present application provides a computer storage medium, comprising computer instructions, when the computer instructions run on an electronic device, the communication device executes the voice interaction processing method in any possible implementation of any aspect.

[0033] In a seventh aspect, the present application provides a computer program product, when the computer program product runs on a computer, the computer executes the voice interaction processing method in any possible implementation of any aspect. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A system architecture schematic diagram is provided for the embodiments of the present application;

[0035] Figure 2 A structure schematic diagram of an electronic device is provided for the embodiments of the present application;

[0036] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0037] Figure 4 A schematic diagram illustrating the principle of a voice interaction processing method provided in an embodiment of this application;

[0038] Figures 5A-5B A schematic diagram illustrating the principle of another voice interaction processing method provided in this application embodiment;

[0039] Figure 6 A schematic diagram illustrating the principle of a telephone call scenario provided in an embodiment of this application;

[0040] Figures 7A-7B A schematic diagram of a scenario for a voice interaction processing method provided in an embodiment;

[0041] Figures 8A-8D A set of application interface diagrams provided for embodiments of this application;

[0042] Figure 9 This is a flowchart illustrating a voice interaction processing method provided in an embodiment of this application. Detailed Implementation

[0043] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " represents "or," for example, A / B can represent A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0044] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying relative importance or implicitly indicating the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this application, unless otherwise stated, "multiple" means two or more. The terms "middle," "left," "right," "upper," and "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0045] In the embodiments of this application, Figure 1 A schematic diagram of a scenario for a voice interaction system 10 according to an embodiment of the present invention is shown.Figure 1 As shown, the system 10 includes an electronic device 100 and a cloud server 200. It should be noted that, Figure 1 The system 10 shown is only an example, and those skilled in the art can understand that in actual applications, the system 10 usually includes a plurality of electronic devices 100 and cloud servers 200, and the number of electronic devices 100 and cloud servers 200 included in the system 10 is not limited by the present application.

[0046] The electronic device 100 is a smart device with voice interaction function, which can receive voice instructions issued by a user and return voice or non-voice information to the user. In the embodiments of the present application, the electronic device 100 can be a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a virtual reality device, a PDA (Personal Digital Assistant, also known as a palmtop computer), a portable Internet device, a data storage device, a camera, a wearable device (for example, a wireless earphone, a smart watch, a smart bracelet, smart glasses, a head-mounted display (HMD), electronic clothing, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, and a smart mirror), or a smart home device (for example, a smart speaker, a smart refrigerator, a smart table lamp, an electric lamp, a smart television, a smart microwave oven, a smart fan, an air conditioner, a smart robot, a smart curtain), and the like. One application scenario involved in the embodiments of the present application is a home scenario, that is, the electronic device 100 is placed in the user's home, and the user can issue voice instructions to the electronic device 100 to achieve certain functions, such as surfing the Internet, playing songs on demand, shopping, understanding weather forecasts, controlling other smart home devices in the home, and the like.

[0047] The cloud server 200 communicates with the electronic device 100 through a network, which can be a cloud server physically located in one or more places. The cloud server 200 provides recognition services for voice data received on the electronic device 100 to obtain a text representation of the voice data input by the user; the cloud server 200 also obtains a representation of the user's intention based on the text representation and generates a response instruction, which is returned to the electronic device 100. The electronic device 100 performs corresponding actions according to the response instruction to provide corresponding services for the user, such as setting an alarm, making a call, sending an email, broadcasting information, playing a song, a video, and the like. Of course, the electronic device 100 can also output a corresponding voice response to the user or display corresponding text content according to the response instruction, and the embodiments of the present application do not limit this.

[0048] Firstly, an electronic device 100 involved in embodiments of the present application is introduced as follows.

[0049] Referring to Figure 2 , Figure 2 A structural schematic diagram of an exemplary electronic device 100 provided by embodiments of the present application is shown.

[0050] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0051] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0053] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0054] The memory in the processor 110 can also be configured to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. Avoiding repeated access reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0055] In some embodiments, the processor 110 can include one or more interfaces. The interface can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0056] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can contain multiple sets of I2C bus. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces respectively. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface, and the touch function of the electronic device 100 is realized.

[0057] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can contain multiple sets of I2S bus. The processor 110 can be coupled to the audio module 170 through the I2S bus, and communication between the processor 110 and the audio module 170 is realized. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the I2S interface, and the function of answering a phone through a Bluetooth headset is realized.

[0058] The PCM interface can also be used for audio communication, which samples, quantizes and encodes analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled through the PCM bus interface. In some embodiments, the audio module 170 can also deliver audio signals to the wireless communication module 160 through the PCM interface, and the function of answering a phone through a Bluetooth headset is realized. Both the I2S interface and the PCM interface can be used for audio communication.

[0059] The UART interface is a universal serial data bus, which is used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface, and the Bluetooth function is realized. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the UART interface, and the function of playing music through a Bluetooth headset is realized.

[0060] The MIPI interface can be used to connect the processor 110 and the display screen 194, the camera 193 and other peripheral devices. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the photographing function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

[0061] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

[0062] The USB interface 130 is an interface that conforms to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other electronic devices, such as AR devices and the like.

[0063] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0064] The charging management module 140 is configured to receive a charging input from a charger.

[0065] The power management module 141 is configured to connect the battery 142 and the charging management module 140 to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be configured to monitor parameters such as the battery capacity, the number of battery cycles, the battery health status (leakage, impedance), and the like. In some embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0066] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0067] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0068] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G and the like applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter, amplify, and the like the received electromagnetic waves, and transmit the processed signals to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor and radiate the signals as electromagnetic waves through the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 can be disposed in the same device.

[0069] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a microphone 170B, etc.), or displays an image or a video through the display 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110 and disposed in the same device as the mobile communication module 150 or other functional modules.

[0070] The wireless communication module 160 can provide a wireless communication solution including UWB, wireless local area networks (WLAN) (e.g., wireless fidelity (WiFi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification, and radiate the signal as an electromagnetic wave via the antenna 2.

[0071] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidu navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0072] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0073] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0074] In some embodiments of the present application, the display screen 194 displays interface content currently output by the system. For example, the interface content is an interface provided by an instant messaging application.

[0075] The electronic device 100 can implement a photographing function through an ISP, the camera 193, a video codec, a GPU, the display screen 194, and an application processor.

[0076] The ISP is configured to process data fed back by the camera 193. For example, when taking a photo, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be disposed in the camera 193.

[0077] The camera 193 is configured to capture still images or videos.

[0078] The digital signal processor is configured to process digital signals. In addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is configured to perform Fourier transform on the frequency point energy, and the like.

[0079] A video codec is used to compress or decompress digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record video in a variety of encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and so on.

[0080] An NPU is a neural-network (NN) computing processor that rapidly processes input information by drawing on the structure of a biological neural network, such as the transmission pattern between neurons in the human brain, and can also constantly self-learn. Through the NPU, the electronic device 100 can implement intelligent cognitive applications, such as image recognition, facial recognition, voice recognition, text understanding, and so on.

[0081] The internal memory 121 can include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs).

[0082] A random access memory can include a static random-access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM, such as a fifth-generation DDR SDRAM commonly referred to as a DDR5 SDRAM), and so on.

[0083] A non-volatile memory can include a disk storage device, a flash memory.

[0084] According to the operation principle, the flash memory can include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. According to the potential order of the storage unit, the flash memory can include single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. According to the storage specification, the flash memory can include universal flash storage (UFS), embedded multi media Card (eMMC), etc.

[0085] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of an operating system or other programs running, and can also be used to store data of users and application programs, etc.

[0086] The non-volatile memory can also store executable programs and data of users and application programs, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110.

[0087] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, music, video, etc. files are saved in the external memory card.

[0088] The electronic device 100 can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0089] The audio module 170 is used to convert digital audio information into analog audio signals, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or part of the function modules of the audio module 170 can be arranged in the processor 110.

[0090] The speaker 170A, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0091] The receiver 170B, also known as the "earpiece", is used to convert the audio electrical signal into the sound signal. When the electronic device 100 answers the phone or voice message, the voice can be answered by placing the receiver 170B close to the human ear.

[0092] The microphone 170C, also known as the "microphone", "sound receiver", is used to convert the sound signal into the electrical signal. When making a phone call or sending a voice message, the user can make a sound by placing the human mouth close to the microphone 170C, and input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, in addition to collecting sound signals, it can also realize the noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, which can realize the functions of collecting sound signals, noise reduction, identifying sound sources, realizing directional recording, etc.

[0093] The earphone interface 170D is used to connect the wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0094] The pressure sensor 180A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The gyroscope sensor 180B can be configured to determine the motion posture of the electronic device 100. The barometric pressure sensor 180C is configured to measure the air pressure. The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of a flip cover. The acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (typically three axes). The acceleration sensor 180E can detect the magnitude and direction of gravity when the electronic device 100 is stationary. The acceleration sensor 180E can also be used to identify the posture of the electronic device 100, for example, to switch between portrait and landscape modes, or as a pedometer. The distance sensor 180F is configured to measure distance. The electronic device 100 can measure distance using infrared or laser. The proximity light sensor 180G can include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is configured to sense the ambient light intensity. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light intensity. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking a picture. The ambient light sensor 180L can also work with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touch. The fingerprint sensor 180H is configured to acquire a fingerprint. The electronic device 100 can use the acquired fingerprint characteristics to implement fingerprint unlocking, access application lock, take pictures using fingerprint, answer incoming calls using fingerprint, and the like. The temperature sensor 180J is configured to detect temperature.

[0095] The touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 together form a touch screen, also referred to as a "touch panel". The touch sensor 180K is configured to detect a touch operation applied to or near the touch sensor 180K. The touch operation refers to an operation in which a user's hand, elbow, stylus, or the like contacts the display screen 194. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. The display screen 194 can provide visual output related to the touch operation. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, which is different from the location of the display screen 194.

[0096] The bone conduction sensor 180M can acquire a vibration signal. In some embodiments, the bone conduction sensor 180M can acquire a vibration signal of a human body sound part vibration bone block. The bone conduction sensor 180M can also contact a human body pulse to receive a blood pressure pulsation signal. In some embodiments, the bone conduction sensor 180M can also be arranged in an earphone to form a bone conduction earphone. The audio module 170 can analyze a voice signal based on the vibration signal of the sound part vibration bone block acquired by the bone conduction sensor 180M to realize a voice function. The application processor can analyze heart rate information based on the blood pressure pulsation signal acquired by the bone conduction sensor 180M to realize a heart rate detection function.

[0097] The keys 190 include a power on key, a volume key, and the like. The keys 190 can be mechanical keys. The keys 190 can also be touch keys. The electronic device 100 can receive a key input to generate a key signal input related to user settings and function control of the electronic device 100.

[0098] The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, playing audio, and the like) can correspond to different vibration feedback effects. Touch operations on different regions of the display screen 194 can also correspond to different vibration feedback effects of the motor 191. Different application scenarios (such as time reminders, received messages, alarms, games, and the like) can also correspond to different vibration feedback effects. The touch vibration feedback effects can also support customization.

[0099] The indicator 192 can be an indicator light that can be used to indicate a charging state, a power change, and can also be used to indicate messages, missed calls, notifications, and the like.

[0100] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the electronic device 100.

[0101] Figure 3 A software structure block diagram of the electronic device 100 of the embodiments of the present application is shown.

[0102] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application program layer, the application program framework layer, the Android runtime and the system library, and the kernel layer.

[0103] The application program layer can include a series of application packages. The application packages can include, for example, applications such as a camera, a gallery, a calendar, a call, a map, navigation, WLAN, Bluetooth, music, video, games, shopping, travel, instant messaging (such as short messages), and the like. In addition, the application packages can also include system applications such as a home screen (i.e., a desktop), a negative one screen, a control center, a notification center, and the like.

[0104] As shown in FIG. 1, the application program layer includes a voice assistant and a voice processing module in an embodiment of the present application. Figure 3

[0105] The voice processing module provides a voice processing capability, and any application program can call the voice processing capability of the voice processing module, for example, an application program of the voice assistant. The electronic device 100 receives a voice signal through the voice assistant application, and the voice assistant application calls the voice processing module to process the voice signal. The voice processing module includes, for example, a capability of automatic speech recognition (ASR), a capability of natural language understanding (NLU), a capability of dialog management (DM), a capability of natural language generation (NLG), and a capability of text to speech (TTS), and the like. Among them,

[0106] The voice recognition module is configured to recognize the voice signal to obtain text representation information of the voice signal. Specifically, the voice recognition module can first represent the voice signal as text data, and then perform word segmentation processing on the text data to obtain the text representation information of the voice signal, that is, convert the vocabulary in the voice signal into input readable by the electronic device 100, including, for example, binary encoding, character sequence, and the like. Typical voice recognition methods can include, for example, a method based on a voice channel model and voice knowledge, a template matching method (comparing the feature vector of the input voice signal with each template in the template library in sequence to output the one with the highest similarity as the recognition result), and a method using a neural network, and the like. The embodiments of the present application do not limit the voice recognition method used for voice recognition processing.

[0107] The semantic understanding module is configured to convert the text representation information of the voice signal into semantic information that can be understood by the electronic device 100. The semantic information includes entities, triples, intents, events, and the like. With this information, the electronic device 100 can understand the language of the user and determine what the user wants to do.

[0108] ​The dialogue management module is configured to determine, based on the semantic information, a next action to be performed by the electronic device 100, the next action including one or more of the following: playing a voice reply content (e.g., providing a result, asking for a specific constraint condition, clarifying or confirming a requirement, etc.); displaying a text content of the voice reply content; jumping to a corresponding interface; and the like.

[0109] Specifically, the dialogue management module determines an intent expressed in the semantic information, and then fills a slot corresponding to the intent according to the semantic information. The intent is what the user wants to do, and the slot corresponding to the intent is information needed by the user to complete the intent, for example, the intent is "making a call", and the slot corresponding to the intent "making a call" is who to call, i.e., the object of the call; for another example, the intent is "sending a message", and the slot corresponding to the intent "sending a message" has two, which are the object of the message and the content of the message.

[0110] In essence, dialogue management is a decision-making process, and the dialogue management module determines the next action to be performed in the voice interaction process according to the current state, thereby assisting the user to complete the task of information acquisition or service acquisition. If the action needs to interact with the user, the natural language generation module will be triggered to generate language text understandable by the user; finally, the generated language text is played to the user by the speech synthesis module.

[0111] The natural language generation module is configured to convert a non-language format data set into a language format text information understandable by the user. The natural language generation module determines which information should be included in the language text being constructed, and organizes a reasonable text order to merge multiple information into a sentence. Then some conjunctions, phrases are selected to form a complete sentence with a good structure.

[0112] The speech synthesis module is configured to convert the text information generated by the natural language generation module into artificial speech by mechanical and electronic methods.

[0113] The application framework layer provides an application programming interface (API) and a programming framework for the application in the application layer. The application framework layer includes some pre-defined functions.

[0114] As shown in FIG. 1, Figure 3 the application framework layer can include an input manager, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a display manager, an activity manager, and the like. For ease of illustration, Figure 3In some embodiments, the application framework layer includes an input manager, a window manager, a content provider, a view system, and an activity manager. It should be noted that any two of the input manager, the window manager, the content provider, the view system, and the activity manager can call each other.

[0115] The input manager is configured to receive instructions or requests from lower layers, such as the kernel layer and the hardware abstraction layer.

[0116] The window manager is configured to manage window programs. The window manager can obtain the size of a display screen, determine whether there is a status bar, lock a screen, and capture a screen. In some embodiments, the window manager is configured to display a window including one or more shortcut controls when the electronic device 100 meets a preset triggering condition.

[0117] The activity manager is configured to manage activities that are running in the system, including process, application, service, and task information.

[0118] The content provider is configured to store and obtain data, and make the data accessible to applications. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, and a phonebook.

[0119] The view system includes visual controls, such as a control for displaying text and a control for displaying images. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying images. In some embodiments, the view system is configured to display a shortcut area on the display screen 103 when the electronic device 100 meets a preset triggering condition. The shortcut area includes one or more shortcut controls added by the electronic device 100. The position and layout of the shortcut area, and the icons, positions, layouts, and functions of the controls in the shortcut area are not limited in the present application.

[0120] The display manager is configured to transmit display content to the kernel layer.

[0121] The telephony manager is configured to provide communication functions of the electronic device 100. For example, the telephony manager is configured to manage a call state (including call connection and call hang-up).

[0122] The resource manager is configured to provide various resources for applications, such as localized strings, icons, images, layout files, and video files.

[0123] The notification manager enables applications to display notification information in the status bar, which can be used to convey alert-type messages that can automatically disappear after a brief stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the form of a figure or a scroll bar text in the system top status bar, such as a notification of a background running application, and can also be a notification that appears in the form of a dialog window on the screen. For example, the status bar prompts text information, issues a prompt sound, the electronic device vibrates, the indicator light flashes, etc.

[0124] The Android Runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0125] The core library contains two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0126] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java file of the application layer and the application framework layer into a binary file. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection, etc.

[0127] The system library can include multiple functional modules. For example: a surface manager, media libraries, a three-dimensional graphics processing library (for example: OpenGL ES), a 2D graphics engine (for example: SGL), etc.

[0128] The surface manager is used to manage the display subsystem, and provides a fusion of 2D and 3D layers for multiple applications.

[0129] The media library supports multiple commonly used audio, video format playback and recording, and static image files, etc. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0130] The three-dimensional graphics processing library is used to realize three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc.

[0131] The 2D graphics engine is a drawing engine for 2D drawing.

[0132] The kernel layer is a layer between hardware and software. The kernel layer at least contains display drivers, camera drivers, audio drivers, sensor drivers, touch chip drivers, and input systems, etc. For ease of illustration, Figure 3In the middle, the kernel layer is taken as an example to illustrate the input system, the driving of the touch chip, the display driving and the storage driving. The display driving and the storage driving can be arranged in the driving module.

[0133] It can be understood that the structure illustrated in the present application does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 can include more or less components than the illustration, or combine certain components, or split certain components, or different component arrangement. The illustrated components can be implemented in hardware, software or a combination of software and hardware.

[0134] The technical principle based on the voice interaction system 10 will be introduced below. As shown in Figure 4 , Figure 4 A voice interaction process in the voice interaction system 10 is shown in the middle. The cloud server 200 and the electronic device 100 communicate through the network.

[0135] First, the electronic device 100 detects that there is a voice signal access, and the electronic device 100 starts the voice interaction function. In some embodiments, the electronic device can receive the first voice signal through the voice assistant application (APP). The electronic device 100 detects whether the received voice signal contains a target object (the target object is, for example, a pre-set wake-up word). If the target object is contained, the interaction state is entered, and the voice interaction function is started. The target object can be pre-set when the electronic device 100 is manufactured, can be pre-set in the voice assistant application, or can be set by the user during the use of the electronic device 100. The length and content of the target object are not limited in the present application.

[0136] Electronic device 100 controls the distribution of received voice signals based on preset rules, with distribution paths including Path 1 and Path 2. These preset rules include: when network quality is good, electronic device 100 uploads the received voice signals to cloud server 200 for processing (Path 1), where good network quality means that electronic device 100 and cloud server 200 can transmit data (including uplink and downlink data transmission); when network quality is poor or disconnected, electronic device 100 processes the received voice signals on its own (Path 2), where poor network quality or disconnection means that electronic device 100 and cloud server 200 cannot transmit data (including uplink or downlink data transmission), or the data transmission rate is below a threshold. The preset rules can also be based on the intent corresponding to the recognized voice signal. Simply put, if the intent corresponding to the voice signal can be completed locally, such as making a phone call, sending a text message, or opening a photo gallery, it can be processed on electronic device 100; if the intent requires network access, such as searching for web pages or playing music online, it can be processed on cloud server 200.

[0137] The following sections describe the processing of voice signals on the cloud server 200 and the processing of voice signals on the electronic device 100.

[0138] Path 1: Process the voice signal on cloud server 200.

[0139] Step 1: Electronic device 100 uploads the voice signal to cloud server 200. Cloud server 200 receives the voice signal and uses ASR speech recognition technology to recognize the voice signal and convert it into text representation information. That is, the words in the voice signal are converted into input that cloud server 200 can read, including, for example, binary encoding, character sequences, etc.

[0140] In some embodiments, the cloud server 200 can sequentially compare the feature vector of the input speech signal with each template in the template library for similarity, take the one with the highest similarity as the recognition result, output text data, and then perform word segmentation on the text data to obtain the text representation information of the speech signal. Optionally, the cloud server 200 can also use a trained vocal tract model, neural network model, etc. to calculate the text representation information corresponding to the speech signal.

[0141] It should be noted that when the cloud server 200 recognizes speech signals using speech recognition technology, it can also perform some preprocessing operations on the speech signals, such as sampling, quantization, removing speech data that does not contain speech content (such as silent speech data), framing the speech data, windowing, and so on.

[0142] Step 2: After speech recognition, the cloud server 200 uses semantic understanding technology (NLU) to convert the text representation information into semantic information that the machine can understand.

[0143] In some embodiments, the execution of semantic understanding technology can be simply understood as the following steps: First, the cloud server 200 segments the text representation information obtained from speech recognition into a series of semantic and syntactic units, usually represented by the word "token". A common text segmentation method is "word segmentation," which segments text at the granularity of "words". Models used for word segmentation can include first-order Markov models, hidden Markov models, conditional random fields, recurrent neural networks, etc.

[0144] Then, based on the token sequence, text representation models such as word vector space models and distributed representation models are used to obtain a numerical vector or matrix. This matrix is ​​the numerical representation of the text. Next, based on the numerical representation of the text, classification algorithms, sequence labeling methods, and other techniques are used to calculate the "key information" (i.e., semantic information), such as entities, triples, intents, events, etc. With this information, the cloud server 200 can understand the user's language and determine what the user wants to do.

[0145] Step 3: The cloud server 200 performs dialogue management based on semantic information. Dialogue management refers to the process by which the cloud server 200 determines the next action to be performed based on semantic information. The actions performed include one or more of the following: playing voice response content (e.g., providing results, inquiring about specific restrictions, clarifying or confirming requirements, etc.); displaying the text content of the voice response; navigating to the corresponding interface; etc.

[0146] In some embodiments, the cloud server 200 determines the intent expressed in the semantic information and then fills the slots corresponding to the intent based on the semantic information. An intent is what the user wants to do, and the corresponding slot is the information the user needs to complete that intent. One intent can correspond to one or more slots. The cloud server 200 fills the slots with the intent based on the semantic information. If insufficient semantic information results in missing information in one or more slots, the next action is determined to be to further query the missing slots. If the information in the slots is not missing, the user intent is converted into a clear instruction from the user, instructing the electronic device 100 to perform the corresponding action.

[0147] For example, cloud server 200 receives the semantic information of the voice signal "Please open my photo library." Based on this semantic information, it determines that the user's intent is to open an object. The slot corresponding to this intent is the object to be opened. Cloud server 200 fills the slot according to the semantic information of the voice signal, determining that the object to be opened is the photo library. Then, the dialogue management determines a clear instruction based on this semantic information, namely, the instruction to open the photo library.

[0148] The above provides an example of how cloud server 200 determines operation commands based on the user's voice signals when no slot information is missing. However, if insufficient semantic information leads to the absence of one or more slots, cloud server 200 needs to save the current intent and slot information and conduct further queries for the missing slots. This is typically referred to as a multi-turn dialogue.

[0149] For example, cloud server 200 obtains the semantic information of the voice signal "I want to make a call". Based on the semantic information of the voice signal, it determines that the user's intention is to make a call. The slot corresponding to this intention is the target to make the call. Due to insufficient semantic information, the slot is missing. Therefore, the dialogue management determines a clear instruction based on the semantic information, that is, to play the voice reply content and to conduct further inquiries on the missing slot.

[0150] Optionally, dialogue management can determine another explicit instruction based on the semantic information of the voice signal, namely, to display the text content of the voice response.

[0151] When the cloud server 200 receives a voice signal again, it repeats steps ① and ② above. Then, based on the saved intent and slot information, it fills the missing slots using the semantic information corresponding to the voice signal. If all slots are filled, meaning no slot information is missing, the user intent is converted into a clear user instruction, directing the electronic device 100 to perform the corresponding action. In the example above, the cloud server 200 saves the intent as "make a phone call," and the missing slot is the target of the call. Therefore, when the cloud server 200 receives the voice signal "Xiaoming" again, it fills the slot "target of the call" based on the semantic information corresponding to the voice signal "Xiaoming," thus determining the instruction to call Xiaoming.

[0152] Step 4: After the cloud server 200 determines the next action to be performed, if the action requires voice interaction with the user, such as outputting voice reply content, the cloud server 200 can generate user-understandable language text based on natural language generation technology, and then synthesize the generated language text into speech data.

[0153] In some embodiments, the cloud server 200 determines which information should be included in the language text being constructed and organizes the text in a logical order, merging multiple pieces of information into a single sentence. Then, it selects conjunctions and phrases to combine this information into a well-structured, complete sentence.

[0154] Step 5: The cloud server 200 sends instructions to the electronic device 100, directing the electronic device 100 to perform the action.

[0155] In some embodiments, based on steps ①②③④ above, the cloud server 200 sends an instruction with voice data (voice reply content) to the electronic device 100, instructing the electronic device 100 to output the voice reply content.

[0156] Optionally, the cloud server 200 sends an instruction containing text data (the text content of the voice response) to the electronic device 100, instructing the electronic device 100 to display the text data.

[0157] In some embodiments, step ④ is optional. If the next action determined by the cloud server 200 does not require outputting a voice response, then step ④ does not need to be executed. Based on the above steps ①②③, the cloud server 200 sends an instruction to the electronic device 100, instructing the electronic device 100 to perform a page transition.

[0158] Path 2: Process the voice signal on electronic device 100.

[0159] Electronic device 100 receives a voice signal, recognizes it using speech recognition technology, and converts it into text representation information. Then, it uses semantic understanding technology to convert the text representation information into machine-understandable semantic information. Next, based on the semantic information, electronic device 100 determines the next action to be performed. If the action requires voice interaction with the user, such as outputting a voice reply, electronic device 100 can generate user-understandable language text using natural language generation technology, and then synthesize the generated language text to generate voice data. Electronic device 100 outputs this voice data. If the next action determined by electronic device 100 does not require outputting a voice reply, then speech synthesis technology is not used, and electronic device 100 performs a page navigation.

[0160] It should be noted that, based on the same inventive concept, Figure 4 The principles of speech recognition, semantic understanding, dialogue management, and speech synthesis in path two of the illustrated embodiment are similar to those in path one. Therefore, the implementation process of speech recognition, semantic understanding, dialogue management, and speech synthesis in path two of the electronic device 100 can be referred to the corresponding descriptions of steps ①②③④⑤ in path one of the cloud server 200, which will not be repeated here.

[0161] In summary, Figure 4 The illustrated embodiments detail the implementation principle of the voice interaction system. In some cases, electronic device 100 distributes voice services based on network conditions. If the network conditions are good, the voice signal is uploaded to cloud server 200 for processing, i.e., path two mentioned above; if the network is disconnected or the network quality is poor, the voice signal is processed on electronic device 100, i.e., path one mentioned above. In this case, if the voice service switches paths during processing due to network issues, the original voice service cannot continue to execute, affecting the user experience.

[0162] For example, when cloud server 200 receives the first voice signal, if the semantic information of the voice signal is insufficient, resulting in the absence of one or more slots, cloud server 200 needs to save the current intent and slot information and perform further queries for the missing slots. In this type of voice service requiring multi-turn dialogue, if a network interruption occurs before cloud server 200 receives the next voice signal, cloud server 200 will be unable to receive the next voice signal. Electronic device 100 will then distribute the next voice signal to itself for processing. Based on the semantic information of this next voice signal, electronic device 100 cannot continue executing the original voice service, resulting in the interruption of the original voice service and impacting user experience.

[0163] In conjunction with the voice interaction system 10 of this application embodiment, this application embodiment also provides a voice interaction processing method. When the cloud server 200 sends an instruction to the electronic device 100 to instruct the electronic device 100 to perform a corresponding action, it simultaneously sends the context of the voice dialogue (intent and slot information) to the electronic device 100. If a network interruption occurs in the voice service of a multi-turn dialogue, resulting in end-to-cloud switching (switching between the electronic device 100 and the cloud server 200, i.e., switching between path one and path two), the electronic device 100 can also continue to execute the original voice service based on the context of the voice dialogue and the received next voice signal, thereby solving the problem of interruption of the voice service in multi-turn dialogue.

[0164] The following details the steps and flow of a voice interaction processing method provided in this application, such as... Figure 5A As shown, Figure 5A The figure shows a voice interaction process in the voice interaction system 10.

[0165] At time T1, when the network quality is good, electronic device 100 receives voice message 1 and initiates the voice interaction function. Electronic device 100 distributes the received voice message 1 according to preset rules. For example, if the network quality is good, electronic device 100 uploads the received voice message 1 to cloud server 200 for processing. The processing includes speech recognition, semantic understanding, dialogue management, and speech synthesis. In this embodiment, voice message 1 can also be referred to as the first voice signal.

[0166] Among them, those based on the same inventive concept Figure 5A The principles of speech recognition, semantic understanding, dialogue management, and speech synthesis for problem-solving at time T1 in the illustrated embodiment are as follows: Figure 4 The path in the illustrated embodiment is similar to that in the first embodiment. Therefore, the implementation process of speech recognition, semantic understanding, dialogue management, and speech synthesis of the cloud server 200 at time T1 can be referred to... Figure 4 The corresponding descriptions of steps ①②③④ in the above path one for Zhongyun Server 200 will not be repeated here.

[0167] The cloud server 200 determines the next action to be performed and sends an instruction to the electronic device 100 to instruct the electronic device 100 to perform action 1. Simultaneously, the cloud server 200 sends a voice dialogue context to the electronic device 100. This dialogue context refers to the intent and slot information obtained by the cloud server 200 through recognition and understanding of voice 1. Action 1 includes one or more of the following: playing a voice response to voice 1 (e.g., providing a result, inquiring about specific limitations, clarifying or confirming a need, etc.); displaying the text content of the voice response; navigating to the corresponding interface; etc.

[0168] After receiving the instruction and dialogue context, the electronic device 100 forwards the instruction and dialogue context through the dialogue information forwarding module. Based on the instruction, the electronic device 100 executes action 1 and saves the dialogue context. The dialogue information forwarding module can be viewed as a node that receives data sent by the cloud server 200, used to receive and forward data.

[0169] At time T2, network quality is poor. After electronic device 100 outputs its voice response to voice 1, it receives voice 2. Due to the poor network quality, data transmission between electronic device 100 and cloud server 200 is impossible, so electronic device 100 cannot upload voice 2 to cloud server 200. Electronic device 100 then utilizes its own voice processing capabilities to process voice 2, including speech recognition, semantic understanding, dialogue management, and speech synthesis. The implementation process of speech recognition, semantic understanding, dialogue management, and speech synthesis by electronic device 100 at time T2 can be found in [reference needed]. Figure 4The corresponding description of the electronic device 100 in path two above will not be repeated here. In this embodiment, voice 2 can also be referred to as the second voice signal.

[0170] It is important to note that, unlike path two described above, in this embodiment, during the dialogue management part, the electronic device 100 determines the next action to be performed based on voice 2 and the dialogue context saved at time T1. The electronic device 100 fills in missing slots based on the semantic information corresponding to voice 2, as well as the intent and slot information. If the semantic information corresponding to voice 2 is insufficient, the slot information is not fully filled, and one or more slots are missing, the electronic device 100 determines that the next action to be performed is to further query the missing slots; if the slot information is not missing, the user's intent is converted into a clear user instruction, instructing the electronic device 100 to perform the corresponding action.

[0171] Here, time T2 refers to the period between when electronic device 100 receives the instruction and dialogue context sent by cloud server 200 for voice 1 and when electronic device 100 uploads voice 2 to cloud server 200. For example, it could be before electronic device 100 receives voice 2, or it could be after receiving voice 2 but before uploading it to cloud server 200. That is, due to poor network quality at time T2, voice 2 cannot be uploaded to cloud server 200.

[0172] In some embodiments, time T2 can also be after the electronic device 100 uploads voice 2 to the cloud server 200, but before the cloud server 200 sends the instruction to the electronic device 100. That is, due to poor network quality, the cloud server 200 is unable to send the instruction generated for voice 2 to the electronic device 100. Figure 5B As shown, electronic device 100 uploads voice 2 to cloud server 200. Cloud server 200 processes voice 2, including speech recognition, semantic understanding, dialogue management, and speech synthesis. At this point, a network quality problem occurs, and data transmission cannot be achieved between electronic device 100 and cloud server 200. Cloud server 200 is unable to send instructions generated for voice 2 to electronic device 100.

[0173] Optionally, if the electronic device 100 does not receive an instruction from the cloud server 200 for the voice 2 within a preset time after the electronic device 100 uploads the voice 2 to the cloud server 200, the electronic device 100 may use its own voice processing capabilities to process the voice 2 (e.g., a backup of the voice 2).

[0174] Optionally, after the electronic device 100 uploads voice 2 to the cloud server 200, if it detects that the network connection with the cloud server 200 has been disconnected before receiving the instruction issued by the cloud server 200 for voice 2, the electronic device 100 can call its own voice processing capability to process voice 2 (for example, a backup of voice 2).

[0175] The processing procedure can be referred to above. Figure 5A The corresponding description of voice 2 by electronic device 100 at time T2 will not be repeated here.

[0176] In this way, during voice interaction, the cloud server 200 sends the dialogue context to the electronic device 100 every time it issues a command. The electronic device 100 receives and saves the dialogue context. In the event of a network interruption, the voice service that was originally being processed on the cloud server 200 can still be processed by the electronic device 100 based on the saved dialogue context, so that the voice service is not interrupted, the processing efficiency of the voice service is improved, and the user experience is enhanced.

[0177] In some embodiments, the cloud server 200 will only send the dialogue context, i.e. the intent and slot information, to the electronic device 100 when each instruction is issued, provided that at least one slot information is missing from one or more slot information.

[0178] Specifically, in combination Figure 5A In the process of processing voice 1 at time T1, cloud server 200 determines the intent expressed by the semantic information of voice 1 and the corresponding slot information based on the semantic information. One intent can correspond to one or more slots. Cloud server 200 fills the slot information of the intent based on the semantic information. If all slots are filled completely, that is, if no slot information is missing, the user intent is converted into a clear instruction from the user. Cloud server 200 then sends the instruction to electronic device 100, instructing electronic device 100 to perform the corresponding action. For example, if cloud server 200 receives the voice signal "Please help me open the gallery," it determines the user's intent as "open" an object based on the semantic information of the voice signal. The slot corresponding to this intent is the object to be opened. Cloud server 200 fills the slot based on the semantic information of the voice signal and determines that the object to be opened is the gallery. Therefore, the dialogue management determines a clear instruction based on this semantic information, namely, the instruction to open the gallery. As can be seen, since the user's intent has been completed at this time, the cloud server 200 determines that the intent has ended. Therefore, the cloud server 200 does not need to send the dialogue context (intent and slot information) to the electronic device 100, thus saving resources.

[0179] In cases where insufficient semantic information leads to the absence of one or more slots, the cloud server 200 needs to save the current intent and slot information, further query the missing slots, and upon receiving the next voice signal, combine the saved intent and slot information to fill the slot using the next voice signal, thus determining the next action. In this embodiment, the cloud server 200 generates a voice response for voice 1 based on speech synthesis technology, sends an instruction to the electronic device 100 to output the voice response, and simultaneously sends a dialogue context (the intent and slot information corresponding to voice 1) to the electronic device 100. The electronic device 100 receives and saves this dialogue context. Thus, even if a network interruption occurs when the electronic device 100 receives the next voice signal, the electronic device 100 can still process the received next voice signal using its own voice interaction capabilities combined with the saved dialogue context, improving the processing efficiency of voice services and enhancing the user experience.

[0180] In some embodiments, when two or more slots are missing, the cloud server 200 synchronously sends not only the intent and slot information to the electronic device 100, but also labels the slots to indicate the order in which the electronic device 100 fills the slots. This way, when the electronic device processes the next voice signal, it can accurately fill one of the slots.

[0181] Next, taking the application scenario of making a phone call as an example, the voice interaction processing method implemented in the phone call scenario in this application embodiment will be described in detail.

[0182] like Figure 6 As shown, at time T1, the network quality is good. When a user wants to make a phone call via voice, they can launch the voice assistant application (APP) and input the voice signal "I want to make a phone call". The electronic device 100 receives the user's voice signal "I want to make a phone call" through the voice assistant application and distributes the received voice signal according to preset rules. For example, if the network quality is good at this time, the electronic device 100 will upload the received "I want to make a phone call" to the cloud server 200 for processing.

[0183] After receiving the voice signal "I want to make a phone call," the cloud server 200 converts the voice signal into text information using Automatic Speech Recognition (ASR) technology and obtains semantic information using Natural Language Understanding (NUL) technology, recognizing the user's intent as making a phone call. Next, the cloud server 200 determines that the slot information corresponding to this intent includes the recipient of the call. The cloud server 200 fills the slot based on the semantic information. However, if the cloud server 200 determines that the semantic information of "I want to make a phone call" does not include the recipient of the call, then the cloud server 200 identifies a gap in the slot information corresponding to this intent (making a phone call).

[0184] The cloud server 200 determines that the next action is to query the vacant slot information. Using text-to-speech (TTS) technology, the cloud server 200 generates a voice response, "Who do you want to call?", and sends an instruction containing this voice response to the electronic device 100, instructing the electronic device 100 to play the voice response. Simultaneously, the cloud server 200 sends the dialogue context to the electronic device 100, which includes the intent "to make a call" and the slot information "the person to call (vacant)". Upon receiving the instruction and dialogue context from the cloud server, the electronic device 100 plays the voice response "Who do you want to call?" based on the instruction and saves the dialogue context.

[0185] Optionally, the cloud server 200 can also send an instruction to the electronic device 100 containing text data with the voice reply content, instructing the electronic device 100 to display the text data (the text content of "Who do you want to call").

[0186] After electronic device 100 plays the voice reply "Who do you want to call?", the user inputs the voice signal "Call Xiaoming" again. At time T2, due to poor network quality, electronic device 100 utilizes its own voice processing capabilities to process the voice signal "Call Xiaoming". Electronic device 100 converts the voice signal into text information using Automatic Speech Recognition (ASR) technology and obtains semantic information using Natural Language Understanding (NUL) technology. Next, electronic device 100 fills the slot based on the saved intent "make a call" and the slot information "the person to call (vacant)", as well as the semantic information corresponding to "Call Xiaoming". Electronic device 100 recognizes "Xiaoming" in the semantic information of "I want to make a call" as the person to call, meaning the cloud server 200 determines that the slot information (the person to call) corresponding to this intent (make a call) is "Xiaoming".

[0187] Electronic device 100 determines that the next action to be performed is to call Xiaoming, and outputs the voice response "Calling Xiaoming". Electronic device 100 generates the voice response "Calling Xiaoming" using text-to-speech (TTS) technology, and plays the voice response. Furthermore, electronic device 100 searches for the contact Xiaoming in its address book and invokes its call function to call Xiaoming. Optionally, electronic device 100 can also display the text data of the voice response (the text content of "Calling Xiaoming").

[0188] The above describes the voice interaction processing method in a telephone scenario. Below, using a smartphone as an example of the aforementioned electronic device 100, some voice interaction processes are illustrated in specific scenarios. During voice interaction, if the network quality of electronic device 100 changes from good to poor, the processing of the voice signal is switched from the cloud server 200 to electronic device 100. Since the cloud server 200 distributes the voice dialogue context and stores it on electronic device 100, even if a network interruption occurs during multi-round dialogue, electronic device 100 can still achieve uninterrupted voice service. For example... Figure 7A and Figure 7B As shown, the wake word is set to "Xiaoyi Xiaoyi".

[0189] User: Xiaoyi, I want to make a phone call.

[0190] Smartphone (Electronic Device 100): Who do you want to call?

[0191] User: Xiaoming.

[0192] Smartphone (Electronic Device 100): Okay, calling Xiaoming for you.

[0193] The following is combined Figures 8A-8D Taking the above-mentioned voice dialogue as an example, this application describes the implementation of the voice interaction processing method provided in the embodiments of the present application on the display interface of a smartphone.

[0194] like Figure 8A As shown, Figure 8A A voice interaction interface 801 is shown, which may be, for example, the interface of a voice assistant application. The voice interaction interface 801 includes a status bar 8011 and a function bar 8012.

[0195] The status bar 8011 may include: one or more signal strength indicators 8013 for the wireless network signal, a battery status indicator 8014, and a time indicator 8015. The signal strength indicator 8013 indicates the current network quality (and may also indicate the data transmission rate between the electronic device 100 and the cloud server 200). Figure 8AIn the image, the signal strength indicator 8013 shows a full signal (4 bars), indicating that the current network quality is good.

[0196] The function bar 8012 may include one or more function controls, such as a voice input control 8016. When the electronic device 100 detects a user operation on the voice input control 8016, the electronic device 100 receives a voice signal. For example... Figure 8A In the process, the electronic device 100 receives the voice signal "Xiaoyi Xiaoyi, I want to make a call" and displays it on the voice interaction interface 801.

[0197] like Figure 8B As shown, when electronic device 100 receives the voice signal "Hey Celia, I want to make a call," it can upload the voice signal to cloud server 200 for processing. Based on the instruction returned by cloud server 200, it plays the voice reply "Who do you want to call?" and displays it on the voice interaction interface 802. The voice input control 8016 is transformed into a voice output control 8026, indicating that electronic device 100 is currently outputting voice. In this embodiment, while cloud server 200 returns the instruction, it simultaneously returns the voice dialogue context to electronic device 100, which receives and saves the dialogue context.

[0198] The user continues to input voice, such as Figure 8C As shown, the network quality of electronic device 100 is currently poor, with only two bars remaining on the signal strength indicator 8033. Therefore, electronic device 100 and cloud server 200 cannot transmit data, or the data transmission rate is too low. When electronic device 100 receives the voice signal "Xiaoming," it cannot upload the voice signal to cloud server 200 for processing, or cloud server 200 cannot send instructions to electronic device 100. In this case, electronic device 100 can continue processing the voice signal "Xiaoming" based on the saved dialogue context, playing the voice reply "Okay, calling Xiaoming for you," and displaying it on the voice interaction interface 803. Furthermore, by performing the action of making a phone call, electronic device 100 jumps to the call interface, as shown... Figure 8D As shown, Figure 8D A call interface 804 is shown, which indicates that the current electronic device 100 is calling Xiaoming.

[0199] The above describes an application scenario for a multi-turn dialogue (specifically a two-turn dialogue) voice service. During the dialogue, the network quality of electronic device 100 changes from good to poor, and the processing of the voice signal is switched from cloud server 200 to electronic device 100. Since cloud server 200 distributes the dialogue context of the voice and stores it on electronic device 100, even if a network interruption occurs during the multi-turn dialogue, electronic device 100 can still achieve uninterrupted voice service, thus improving the processing efficiency of the voice service.

[0200] Next, this application embodiment provides another application scenario of a three-turn dialogue, taking the application scenario of sending text messages as an example, to briefly explain the voice interaction processing method implemented in the scenario of sending text messages in this application embodiment.

[0201] When the network quality is good, the electronic device 100 receives the voice signal "I want to send a text message" input by the user. Based on preset rules, the electronic device 100 distributes the received voice signal. For example, if the network quality is good at this time, the electronic device 100 will upload the received "I want to send a text message" to the cloud server 200 for processing.

[0202] The cloud server 200 identifies the user's intent as sending an SMS. Next, the cloud server 200 determines that the slot information corresponding to this intent includes the recipient of the SMS and the content of the SMS. The cloud server 200 fills the slots based on the semantic information. However, the cloud server 200 recognizes that the semantic information for "I want to send an SMS" does not include the recipient of the call or the content of the SMS; that is, the cloud server 200 determines that the information for the slots (recipient of the SMS and content of the SMS) corresponding to this intent (sending an SMS) is vacant.

[0203] The cloud server 200 determines that the next action to be performed is to query for vacant slot information. Since there are two vacant slots, the cloud server 200 can query one of the vacant slots according to priority, for example, first querying the recipient of the text message. The cloud server 200 generates a voice response, "Who do you want to text?", using text-to-speech (TTS) technology, and sends an instruction containing this voice response to the electronic device 100, instructing the electronic device 100 to play the voice response. Simultaneously, the cloud server 200 sends the dialogue context to the electronic device 100, which includes the intent "send text message" and the slot information "recipient of the text message (vacant), content of the text message (vacant)". Upon receiving the instruction and dialogue context from the cloud server, the electronic device 100 plays the voice response "Who do you want to text?" based on the instruction and saves the dialogue context.

[0204] Next, after the electronic device 100 plays the voice reply "Who do you want to send a text message to?", the user inputs the voice signal "To Xiaoming" again. If the network quality is good at this time, the electronic device 100 uploads the received "I want to send a text message" to the cloud server 200 for processing. The cloud server 200 fills the slot based on the saved intent "send text message" and slot information "target of text message (empty), content of text message (empty)" and the semantic information corresponding to "to Xiaoming". The cloud server 200 recognizes that "Xiaoming" in the semantic information of "to Xiaoming" is the target of the text message, that is, the cloud server 200 determines that the information of the slot (target of text message) corresponding to this intent (making a phone call) is "Xiaoming".

[0205] Since the slot information "content of the text message" is still empty at this time, the cloud server 200 saves the current intent and slot information. The cloud server 200 determines that the next action to be performed is to query again for the empty slot information (content of the text message). The cloud server 200 generates a voice reply, "What do you want to send?", based on text-to-speech (TTS) technology, and sends an instruction with this voice reply to the electronic device 100, instructing the electronic device 100 to play the voice reply. Simultaneously, the cloud server 200 sends the dialogue context to the electronic device 100. At this time, the dialogue context includes the intent (sending a text message), the slot information "recipient of the text message (Xiaoming), content of the text message (empty)". Upon receiving the instruction and dialogue context from the cloud server, the electronic device 100 plays the voice reply "What do you want to send?" based on the instruction and saves the dialogue context.

[0206] In some embodiments, after the electronic device 100 plays the voice reply "Who do you want to send a text message to?", the user inputs the voice signal "To Xiaoming" again. If the network quality is poor at this time, the electronic device 100 uses its own voice processing capability to process the voice signal "To Xiaoming". The electronic device 100 converts the voice signal into text information using Automatic Speech Recognition (ASR) technology and obtains semantic information using Natural Language Understanding (NUL) technology. Then, the electronic device 100 fills the slot based on the stored intent "send text message" and slot information "recipient of the text message (empty), content of the text message (empty)", and the semantic information corresponding to "To Xiaoming". The electronic device 100 recognizes that "Xiaoming" in the semantic information of "To Xiaoming" is the recipient of the text message, that is, the electronic device 100 determines that the information of the slot (recipient of the text message) corresponding to the intent (making a phone call) is "Xiaoming".

[0207] In some embodiments, when two or more slots are missing, the cloud server 200 synchronously sends not only the intent and slot information to the electronic device 100, but also labels the slots to indicate the order in which the electronic device 100 fills the slots. This way, when the electronic device processes the next voice signal, it can accurately fill one of the slots.

[0208] In the example above, when cloud server 200 synchronously sends intent and slot information to electronic device 100, since there are two empty slots, cloud server 200 can mark the slots to determine which slot will be filled next. Therefore, when electronic device 100 fills the slots, it doesn't need to determine which slot the semantic information corresponds to and can directly fill it. That is, electronic device 100 can directly determine that the slot information (the recipient of the text message) corresponding to the intent (making a phone call) is "Xiaoming".

[0209] Since the slot information "content of the text message" is still empty at this time, the electronic device 100 saves the current intent and slot information. The electronic device 100 determines that the next action to be performed is to query again for the empty slot information (content of the text message). The electronic device 100 generates a voice reply "What do you want to send" based on text-to-speech (TTS) technology, and the electronic device 100 plays the voice reply.

[0210] The electronic device 100 processes and fills the slots again upon receiving the next voice signal until the slot information is completely filled, and generates an instruction to execute the intent. Then, the electronic device 100 determines that the intent has been executed.

[0211] This application provides a voice interaction processing method, such as Figure 9 As shown, the method includes:

[0212] Electronic device 100 establishes a connection with cloud server 200. Step S101: Electronic device 100 receives the first voice signal.

[0213] The first voice signal can be, for example, the one described above. Figure 5A or Figure 5B The voice 1 in the text can also be Figure 6 The voice prompt in the video says, "I want to make a phone call."

[0214] Step S102: Electronic device 100 uploads the first voice signal to cloud server 200.

[0215] Step S103: The cloud server 200 recognizes the first voice signal, obtains the corresponding intent and one or more slot information corresponding to the intent, and determines the first voice response content based on the intent and one or more slot information.

[0216] Step S104: The cloud server 200 sends the first voice response content, intent, and one or more slot information to the electronic device 100.

[0217] Step S105: Electronic device 100 outputs the first voice response content and saves the intent and one or more slot information.

[0218] The content of the first voice response could be, for example, as described above. Figure 5A The voice response content included in action 1 can also be Figure 6 The voice prompt in the message asks, "Who would you like to call?"

[0219] The communication quality between electronic device 100 and cloud server 200 is poor.

[0220] Step S106: Electronic device 100 receives the second voice signal.

[0221] The first voice signal can be, for example, the one described above. Figure 5A or Figure 5B Voice 2 in the text can also be Figure 6 The voice message in the video says "Call Xiaoming".

[0222] Step S107: Electronic device 100 recognizes the second voice signal, obtains the corresponding semantic information, and determines the first operation based on the intent and one or more slot information and semantic information.

[0223] Step S108: Perform the first operation.

[0224] The first operation could be, for example, Figure 5A or Figure 5B Action 2 in the text can also be Figure 6 The program plays audio content and / or displays the text "Calling Xiaoming", and performs one or more of the following three actions: "Calling Xiaoming".

[0225] In some embodiments, poor communication quality between the electronic device 100 and the cloud server 200 can occur at any time between steps S106 and S107.

[0226] In one possible implementation, the electronic device 100 recognizes the second voice signal to obtain corresponding semantic information, and determines a first operation based on the intent and one or more slot information and semantic information. This includes: the electronic device 100 identifies a missing slot match between the semantic information and one or more slot information, and fills the missing slot with the semantic information; the electronic device determines the first operation based on the intent and the filled slot information. Specifically, this describes the process by which the electronic device processes the original voice service based on the second voice signal. Since the electronic device has obtained the intent and slot information corresponding to the first voice signal, it can continue to perform slot-filling processing on the received second voice signal based on the intent and slot information, thereby enabling it to continue processing the original voice service.

[0227] In one possible implementation, the first operation includes one or more of the following: playing the second voice response content; displaying the text content of the second voice response content; and navigating to the corresponding interface. The second voice response content can be, for example, as described above. Figure 5A or Figure 5B The voice reply content included in action 2 can also be Figure 6 The voice prompt in the video says, "Calling Xiaoming."

[0228] In one possible implementation, the method further includes: the electronic device receiving a first instruction sent by a cloud server; the electronic device displaying the text content of the first voice response based on the first instruction, and / or jumping to the corresponding interface.

[0229] In one possible implementation, poor communication quality between the electronic device and the cloud server includes: the electronic device failing to upload a second voice signal to the cloud server; or the electronic device uploading a first voice signal to the cloud server but not receiving a response from the cloud server within a preset time. This clarifies when the poor communication quality occurs; it could be during the electronic device uploading the second voice signal, or during the cloud server sending a voice response to the second voice signal.

[0230] In one possible implementation, the electronic device receives the first voice signal, including: the electronic device receives the first voice signal through a voice assistant application.

[0231] This application also provides a computer-readable storage medium. The methods described in the above method embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted on a computer-readable medium. A computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any available medium accessible by a computer.

[0232] This application also provides a computer program product. The methods described in the above method embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. If implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device.

[0233] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0234] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A voice interaction processing method, characterized by, The method comprises: The electronic device receives a first voice signal; In the case that the electronic device establishes a connection with the cloud server, the electronic device uploads the first voice signal to the cloud server; The electronic device receives the first voice reply content, the intent and one or more slot information corresponding to the intent sent by the cloud server, the intent and the one or more slot information are obtained by the cloud server identifying the first voice signal, the first voice reply content is determined by the cloud server based on the intent and the one or more slot information, and the first voice reply content, the intent and the one or more slot information are sent by the cloud server in the case that at least one slot information in the one or more slot information is missing; After the electronic device outputs the first voice reply content, it receives a second voice signal, which is used to fill at least one slot information missing in the one or more slot information; In the case that the communication quality between the electronic device and the cloud server is poor, the electronic device identifies the second voice signal to obtain corresponding semantic information, and determines a first operation based on the intent, the one or more slot information and the semantic information; The electronic device executes the first operation.

2. The method of claim 1, wherein, The electronic device determines a first operation based on the intent, the one or more slot information and the semantic information, comprising: The electronic device identifies that the semantic information and one of the one or more slot information are missing, and fills the semantic information as the value of the slot; The electronic device determines a first operation based on the intent and the filled one or more slot information.

3. The method according to claim 1 or 2, characterized in that, The first operation comprises one or more of the following: Play a second voice reply content; Display the text content of the second voice reply content; Jump to the corresponding interface.

4. The method according to claim 1 or 2, characterized in that, The method further comprises: The electronic device receives a first instruction sent by the cloud server; The electronic device displays the text content of the first voice reply content based on the first instruction, and / or jumps to the corresponding interface.

5. The method according to claim 1 or 2, characterized in that, The communication quality between the electronic device and the cloud server is poor, comprising: The electronic device fails to upload the second voice signal to the cloud server; or After the electronic device uploads the first voice signal to the cloud server, it does not receive the reply data of the cloud server within a preset time.

6. The method of claim 1 or 2, wherein, The electronic device receives a first voice signal, comprising: The electronic device receives the first voice signal through a voice assistant application.

7. A voice interaction processing method, characterized by, The method comprises: The cloud server receives a first voice signal uploaded by an electronic device; The cloud server identifies the first voice signal to obtain a corresponding intent and one or more slot information corresponding to the intent, and determines a first voice reply content based on the intent and the one or more slot information; The cloud server sends the first voice reply content, the intent and the one or more slot information to the electronic device in the case that at least one slot information in the one or more slot information is missing, and the first voice reply content, the intent and the one or more slot information are used to make the electronic device: After outputting the first voice reply content, a second voice signal is received, and the second voice signal is used to fill at least one slot information in the one or more slot information which is missing; In the case that the communication quality between the electronic device and the cloud server is poor, the second voice signal is recognized to obtain corresponding semantic information, and a first operation is determined based on the intent, the one or more slot information and the semantic information; The first operation is executed.

8. An electronic device, comprising: Comprise: One or more processors, one or more memories; the one or more memories are respectively coupled with the one or more processors; the one or more memories are used to store computer program codes, the computer program codes comprise computer instructions; when the computer instructions run on the processor, make the electronic device execute: Receive a first voice signal; In the case that a connection is established with the cloud server, the first voice signal is uploaded to the cloud server; Receive the first voice reply content and the intent and the one or more slot information corresponding to the intent sent by the cloud server, the intent and the one or more slot information are obtained by the cloud server by recognizing the first voice signal, the first voice reply content is determined by the cloud server based on the intent and the one or more slot information, and the first voice reply content, the intent and the one or more slot information are sent by the cloud server in the case that at least one slot information in the one or more slot information is missing; After outputting the first voice reply content, a second voice signal is received, and the second voice signal is used to fill at least one slot information in the one or more slot information which is missing; In the case that the communication quality between the electronic device and the cloud server is poor, the second voice signal is recognized to obtain corresponding semantic information, and a first operation is determined based on the intent, the one or more slot information and the semantic information; The first operation is executed.

9. The electronic device of claim 8, wherein, The determination of the first operation based on the intent, the one or more slot information and the semantic information comprises: Identify that one of the semantic information and the one or more slot information is missing, and fill the semantic information as the value of the slot; Determine a first operation based on the intent and the filled one or more slot information.

10. The electronic device of claim 8 or 9, wherein, The first operation comprises one or more of the following: Play a second voice reply content; Display the text content of the second voice reply content; Jump to the corresponding interface.

11. The electronic device of claim 8 or 9, wherein, The electronic device further executes: Receive the first instruction sent by the cloud server; Display the text content of the first voice reply content based on the first instruction, and / or jump to the corresponding interface.

12. The electronic device of claim 8 or 9, wherein, The communication quality with the cloud server is poor, including: The uploading of the second voice signal to the cloud server fails; or After uploading the first voice signal to the cloud server, no reply data from the cloud server is received within a preset time.

13. The electronic device of claim 8 or 9, wherein, The electronic device receives a first voice signal, including: The first voice signal is received through a voice assistant application.

14. A cloud server, characterized by, Including: One or more processors, one or more memories; the one or more memories are respectively coupled with the one or more processors; the one or more memories are used to store computer program codes, the computer program codes include computer instructions; when the computer instructions run on the processor, make the cloud server execute: Receive the first voice signal uploaded by the electronic device; Identify the first voice signal to obtain corresponding intent and one or more slot information corresponding to the intent, and determine the first voice reply content based on the intent and the one or more slot information; In the case that at least one slot information in the one or more slot information is missing, send the first voice reply content, the intent and the one or more slot information to the electronic device, the first voice reply content, the intent and the one or more slot information are used to make the electronic device: After outputting the first voice reply content, receive a second voice signal, the second voice signal is used to fill at least one slot information in the one or more slot information which is missing; In the case that the communication quality between the electronic device and the cloud server is poor, identify the second voice signal to obtain corresponding semantic information, and determine the first operation based on the intent and the one or more slot information and the semantic information; Execute the first operation.

15. A computer readable medium for storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs include instructions for executing the method of any one of claims 1-7.

Images