Intelligent device and intelligent voice interaction method

Abstract

The present application provides an intelligent device and an intelligent voice interaction method, wherein the method can generate a feedback statement according to the voice control audio after the user inputs the voice control audio, and calculate the emotion score of the feedback statement according to the set emotion scoring mechanism, thereby determining the target emotion type according to the emotion score, and finally playing the feedback statement according to the target emotion type to form a voice broadcast. The method can perform weighted summation calculation on the classification scores of multimodal information such as scene classification, feedback statement content, and image recognition results through the feedback statement, obtain the emotion score, and finally determine the target emotion type, so that the intelligent device can play the voice broadcast audio with a more reasonable emotion type, alleviating the problem of poor voice broadcast effect.

Description

Technical Field

[0001] The present application relates to the field of intelligent voice control technology, and in particular to an intelligent device and an intelligent voice interaction method. Background Art

[0002] Intelligent voice control is a new interactive mode that can control the operation of the device by semantically recognizing the voice information input by the user and then generating control instructions based on the semantic recognition results. In order to realize the interactive process based on intelligent voice control, an intelligent voice system can be built into the smart device. The intelligent voice system can be composed of hardware and software parts. Among them, the hardware part mainly includes a microphone, a speaker and a controller for receiving, feedback and processing voice information; the software part mainly includes a voice conversion module, a natural language processing module and a control module, which are used to convert the input sound signal into a text signal and form specific control instructions for control.

[0003] The intelligent voice control system can feedback the voice control results through voice broadcast. For example, when the user inputs the voice "I want to watch ××'s movie", the intelligent voice system will trigger the media resource search or recommendation function, so that the intelligent device will present the media resource recommendation interface and play the voice "We have found ××'s movie for you". Since the content of the voice broadcast is composed of multiple pre-recorded or generated voice clips by the intelligent voice system, when playing the voice broadcast content, there will be playback effects that do not conform to conventional language specifications, such as incorrect punctuation, wrong emotions, etc., which will reduce the user's interactive experience.

[0004] In order to enhance the user's interactive experience, the intelligent voice system can record voice clips in various emotional expressions and play the corresponding voice clips in different control processes, thereby giving corresponding emotions to the voice broadcast process. For example, use broadcast emotions to play voice content in the news category; use reading emotions to play voice content in the literary works category. However, this voice broadcast method is adapted to fewer application scenarios, that is, it can only realize voice broadcast for texts with clear scene labels, and cannot be used in intelligent question and answer and voice control processes, which reduces the voice broadcast effect of the intelligent voice interaction method. Summary of the invention

[0005] The present application provides an intelligent device and an intelligent voice interaction method to solve the problem of poor voice broadcast effect of traditional voice control methods.

[0006] In one aspect, the present application provides a smart device, comprising: a display, an audio input device, an audio output device, and a controller. The display is configured to display a user interface; the audio input device is configured to receive voice control audio input by a user; the audio output device is configured to output voice broadcast audio; and the controller is configured to execute the following program steps:

[0007] Get the voice control audio input by the user;

[0008] In response to the voice control audio, generating a feedback sentence based on the voice control audio;

[0009] Calculating the sentiment score of the feedback sentence, the sentiment score being a weighted sum of classification scores of multimodal information corresponding to the feedback sentence; the multimodal information comprising a combination of one or more of scene classification, feedback sentence content, and image recognition results;

[0010] A target emotion type is determined according to the emotion score, and the feedback sentence is played according to the target emotion type.

[0011] On the other hand, the present application provides an intelligent voice interaction method, which is applied to an intelligent device, and the intelligent voice interaction method comprises the following steps:

[0012] Get the voice control audio input by the user;

[0013] In response to the voice control audio, generating a feedback sentence based on the voice control audio;

[0014] Calculating the sentiment score of the feedback sentence, the sentiment score being a weighted sum of classification scores of multimodal information corresponding to the feedback sentence; the multimodal information comprising a combination of one or more of scene classification, feedback sentence content, and image recognition results;

[0015] A target emotion type is determined according to the emotion score, and the feedback sentence is played according to the target emotion type.

[0016] It can be seen from the above technical solutions that the intelligent device and intelligent voice interaction method provided by the present application can generate feedback sentences according to the voice control audio after the user inputs the voice control audio, and calculate the emotional score of the feedback sentence according to the set emotional scoring mechanism, thereby determining the target emotional type according to the emotional score, and finally playing the feedback sentence according to the target emotional type to form a voice broadcast. The method can perform weighted summation calculation on the classification scores of multimodal information such as scene classification, feedback sentence content, and image recognition results through feedback sentences, obtain emotional scores, and finally determine the target emotional type, so that the intelligent device can play the voice broadcast audio with a more reasonable emotional type, alleviating the problem of poor voice broadcast effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 This is a usage scenario diagram of the smart device in the embodiment of this application;

[0019] Figure 2 A hardware configuration diagram of the smart device in the embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the connection structure of the audio input / output device of the smart device in the embodiment of the present application;

[0021] Figure 4 This is a schematic diagram of the process of synthesizing voice broadcast audio in an embodiment of the present application;

[0022] Figure 5 A flow chart of generating voice broadcast audio data according to scene classification in an embodiment of the present application;

[0023] Figure 6 This is a schematic diagram of the structure of the smart device in the embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the intelligent voice interaction process in the embodiment of this application;

[0025] Figure 8 A schematic diagram of a flow chart of generating feedback statements in a visual question-answering process in an embodiment of the present application;

[0026] Fig. 9 This is a schematic diagram of the emotion scoring process in the embodiment of this application;

[0027] Fig.10 Schematic diagram of the TTS model structure in the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0029] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0030] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0031] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0032] The smart device provided in the embodiments of the present application may be a voice interaction device, including but not limited to: a smart speaker, a voice dialogue robot, and an intelligent question-and-answer system. The smart device may also be a terminal device with integrated intelligent voice interaction function, such as a smart TV, a mobile terminal, a tablet computer, a personal computer, etc. In some embodiments, the smart device may also be an electronic device with voice interaction function by running a local application or a network application, such as a workstation, a smart home system, a smart network system, etc.

[0033] Taking smart display devices such as smart TVs as an example, the smart device 200 can transmit data and control instructions to the control device 100 to achieve human-computer interaction. Figure 1 FIG. 1 is a schematic diagram of an operation scenario between a smart device and a control device according to an embodiment. Figure 1 As shown, the user can operate the smart device 200 through the smart terminal 300 or the control apparatus 100 .

[0034] In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the smart device includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, and the smart device 200 is controlled wirelessly or wired. The user can input user commands through buttons on the remote controller, voice input, control panel input, etc. to control the smart device 200.

[0035] In some embodiments, a smart terminal 300 (such as a mobile terminal, a tablet computer, a computer, a laptop computer, etc.) may also be used to control the smart device 200. For example, the smart device 200 may be controlled using an application running on the smart device.

[0036] In some embodiments, the smart device may not use the above-mentioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0037] In some embodiments, the smart device 200 can also be controlled in ways other than the control device 100 and the smart terminal 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the smart device 200, or the user's voice command control can be received through a voice control device set outside the smart device 200.

[0038] In some embodiments, the smart device 200 also communicates data with the server 400. The smart device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactions to the smart device 200. The server 400 may be one cluster or multiple clusters, and may include one or more types of servers.

[0039] like Figure 2 As shown, in order to realize the voice interaction function, the smart device 200 may include the following hardware configurations: tuner 210, communicator 220, detector 230, external device interface 240, controller 250, display 260, audio output interface 270, memory, power supply, user interface, etc. Depending on the actual functions of the smart device 200, the smart device 200 may include all of the above hardware configurations, or may include some of them. For example, for a smart TV device, all of the above hardware configurations may be included; while for a smart speaker device, the tuner 210, external device interface 240, display 260, and user interface, etc. may not be included.

[0040] In some embodiments, the communicator 220 in the smart device 200 is a component for communicating with an external device or a server 400 according to various communication protocol types. For example, the communicator 220 may include at least one of a WiFi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or a near field communication protocol chip, and an infrared receiver. The smart device 200 may establish transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0041] The detector 230 is used to collect signals from the external environment or interact with the outside. For example, the detector 230 includes an audio input device 231 such as a microphone, which is used to receive external sounds and convert sound signals into audio signals in the form of electrical signals to transmit to other modules for processing to achieve user voice input.

[0042] For the smart device 200 including the display 260, the display 260 may include the following components, namely: a display screen component for presenting images; a driving component for driving image display; a component for receiving an image signal output from the controller 250, displaying video content, image content, and a menu control interface component, and a user control UI interface component, etc. Therefore, the display 260 may be a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0043] The controller 250 controls the operation of the smart device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the smart device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0044] In some embodiments, the controller 250 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0045] For a smart device 200 with a built-in external device interface 240, the external device interface 240 may include, but is not limited to, any one or more of the following interfaces: a high-definition multimedia interface interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by the above multiple interfaces.

[0046] The audio output interface 270 is used to output audio signals, and can be connected to an audio output device 271 built into the smart device 200 or an audio output device 271 external to the smart device 200 , thereby converting audio signals in the form of electrical signals into sound signals in the form of sound.

[0047] like Figure 3As shown, for the audio input device 231 and the audio output device 271. The audio input device 231 and the audio output device 271 can be built-in or externally connected to the smart device 200. That is, in some embodiments, the audio input device 231 can be used as a detector 230 of the smart device 200, such as a sound detector such as a microphone. In other embodiments, the audio input device 231 can be an independent device, connected to the smart device 200 through the external device interface 240.

[0048] Similarly, in some embodiments, the audio output device 271 may be a speaker built into the smart device 200 and connected to the audio output interface 270. In other embodiments, the audio output device 271 is an audio output device such as a speaker or earphone connected to the audio output interface 271 or the external device interface 240.

[0049] Obviously, different connection methods can be used for the audio output device 271 of the external smart device 200 according to different specific interface methods. For example, when the audio output interface 270 is a 3.5mm audio interface, a 3.5mm audio plug is provided on the audio output device 271 to achieve the connection between the audio output device 271 and the smart device 200. For another example, when the external device interface 240 on the smart device 200 is a USB interface, the audio output device 271 can also be connected to the external device interface 240 through the provided USB plug to achieve the connection between the audio output device 271 and the smart device 200.

[0050] In some embodiments, the audio input device 231 and the audio output device 271 of the external smart device 200 can also be connected by wireless communication. For example, the communicator 220 of the smart device 200 has a built-in Bluetooth connection module, and the audio input device 231 and the audio output device 271 are also provided with a Bluetooth connection module, then the audio input device 231 and the audio output device 271 can achieve communication connection with the smart device 200 through the Bluetooth connection module.

[0051] The above-mentioned audio input device 231 and audio output device 271 can be used to receive voice control audio input by the user and output voice broadcast audio, respectively. The smart device 200 with built-in or external audio input device 231 and audio output device 271 can support the user's intelligent voice control. That is, in the process of intelligent voice control, the audio input device 231 can receive the voice input by the user, and convert the sound signal corresponding to the voice into an electrical signal to form a voice control audio signal. The audio input device 231 then sends the generated voice control audio signal to the controller 250. The controller 250 processes, converts, recognizes and other operations on the voice control audio signal according to the control program corresponding to the intelligent voice system to form a specific control instruction. Then, by executing the control instruction, the voice control effect is achieved.

[0052] While executing the control instruction, the controller 250 can also send the voice broadcast audio corresponding to the control instruction to the audio output device 271, and control the audio output device 271 to play the voice broadcast audio. For example, for the smart device 200 including the display 260, when the user inputs the voice "I want to watch ××'s movie" through the audio input device 231, the audio input device 231 converts the voice content into a voice control audio signal and sends it to the controller 250. The controller 250 executes the control program corresponding to the intelligent voice system, and finally converts the voice content of "I want to watch ××'s movie" into a control instruction. The content corresponding to the control instruction can be a media resource recommendation interface with the index word "××". The controller 250 then controls the display 260 to display the media resource recommendation interface by executing the control instruction, such as Figure 3 While displaying the media resource recommendation interface, in order to achieve a voice interaction effect, the controller 250 can also obtain a voice broadcast audio that is compatible with the control instruction, that is, "××'s movie has been found for you". The voice broadcast audio is then played through the audio output device 271.

[0053] It should be noted that the above-mentioned voice interaction process is only used as an example. In actual applications, the intelligent voice interaction function is not only applied to the above-mentioned smart device 200, but also can be applied to other types of smart devices. For example, smart speakers, vehicle-mounted systems, virtual reality devices, smart wearable devices, etc. In order to facilitate the description of the subsequent implementation methods, only the smart device 200 is used as an example for scheme expression. It should be understood that the device that specifically performs intelligent voice interaction in the subsequent implementation methods can also be other types of smart devices. That is, smart devices that can apply the intelligent voice interaction function can receive the voice control audio input by the user through the built-in or external audio input device 231, and process the voice control audio through the built-in controller or processor of the smart device to generate and execute control instructions. At the same time, through the built-in or external audio output device 271, the voice broadcast audio that needs to be output when executing the control instruction is played to complete the intelligent voice control.

[0054] In the process of using the intelligent voice control function, for the intelligent device 200 including the display 260, the intelligent device 200 can present the interaction result to the user by combining the voice broadcast audio and the display screen. The voice broadcast audio can be formed by splicing pre-recorded audio clips. For example, Figure 4 As shown, the voice broadcast audio "A movie of ×× has been found for you" can be formed by splicing three pre-recorded audio segments, namely segment 1: "Found for you", segment 2: "××", and segment 3: "Movie of".

[0055] However, since the audio segments used to splice the voice broadcast audio are recorded independently, the voice broadcast audio formed after the audio splicing will have problems such as unsmooth sentences at the connection of the voice segments, affected by factors such as the tone, intonation, and emotion of the recorder when recording the voice. For example, there are pauses at the connection positions corresponding to segment 1 "Found for you" and segment 2 "××", and segment 2 "××" and segment 3 "Movie", making the voice broadcast effect unsmooth and reducing the user experience.

[0056] In order to adapt to this voice broadcasting method of splicing audio clips, operators of intelligent voice systems generally can only use a calm tone and intonation without emotion when recording audio clips. Voice interaction is designed to imitate the user's normal conversation process. The emotionless voice broadcasting method cannot adapt to a variety of language environments, which affects the user's enthusiasm for using the voice interaction function.

[0057] To this end, in some embodiments, when pre-recording voice clips, voice clips with the same content can be recorded using multiple different tones or emotions. For example, for a voice clip with the content "found for you", a calm, happy, and sad tone can be used to record the audio clips, thereby obtaining three voice clips with the same content but different audio expressions.

[0058] When the intelligent voice system pre-records audio clips with different tones or emotions, the intelligent device 200 can first determine the appropriate emotion type according to the use time, weather, user settings and other conditions, and then extract the voice clips under the emotion type from the voice clip database to form the voice broadcast audio. For example, when the weather is sunny when the user performs voice interaction, it can be determined that the appropriate emotion type is happy. Therefore, after the user enters the voice content of "I want to watch ××'s movie", the intelligent device 200 can extract the emotion type of "happy" in the voice clip database. Segment 1, segment 2 and segment 3.

[0059] It should be noted that the same control command may be expressed differently in different contexts, such as the use of modal particles, interjections, punctuation marks, etc. Therefore, when recording voice clips, you can also record audio with different contents for the same feedback text. For example, for the control command of the media resource recommendation interface that displays the index word "××", the intelligent voice system can record the following separately: "××'s movie has been found for you" in a calm tone; "I found many ××'s movies!" in a happy tone; "××'s movies are as follows" in a sad tone.

[0060] In some embodiments, the specific emotions used in voice broadcasting can be divided into different emotion types according to the application scenario. A mapping relationship can be established between each emotion type and one or more application scenarios. During the voice broadcasting process, the smart device 200 can first determine the current application scenario type, and then determine the emotion classification suitable for the current application scenario based on the established mapping relationship. For example, Figure 5 As shown, when the user needs to make a voice broadcast, the smart device 200 can detect the current display interface content, interface logic, and input voice interaction content, and when it is determined that the current application scenario is news broadcast, the voice content of the news category can be played using the broadcast emotion. When the smart device 200 detects that the current application scenario is literary work reading, the voice content of the literary work category can be played using the reading emotion.

[0061] However, this voice broadcast method is only applicable to a limited number of application scenarios, that is, it can only be used to broadcast text with clear scenario labels, and cannot be used in intelligent question-answering and voice control, which reduces the voice broadcast effect of the intelligent voice interaction method. In addition, due to the influence of factors such as intonation and speech speed, in the process of pre-recording voice clips, the intonation and speech speed of different voice clips are different. Therefore, when combining voice clips with the same emotion, a smooth voice expression effect cannot be obtained, which reduces the user's interactive experience.

[0062] In order to further improve the interactive experience, an intelligent voice interaction method is provided in some embodiments of the present application, and the method can be applied to the smart device 200 to achieve a better interactive experience. Figure 6 As shown, in order to meet the application of the intelligent voice interaction method, the intelligent device 200 should at least include a display 260, an audio input device 231, an audio output device 271 and a controller 250. Among them, the display 260 is configured to display a user interface. The audio input device 231 is configured to receive voice control audio input by the user, and the audio output device 271 is configured to output voice broadcast audio. The controller 250 is configured to execute the application corresponding to the intelligent voice interaction method to realize the intelligent voice interaction function with the user. Specifically, as Figure 7 As shown, the intelligent voice interaction method includes the following contents:

[0063] Acquire the voice control audio input by the user. When using the smart device 200, the user can input the voice control audio in different ways. In some embodiments, a voice function button may be provided on the control device 100, and the user can long press the voice function button to enable the smart device 200 to receive the user's voice control audio. The voice control audio is input during the period of pressing the voice function button.

[0064] Obviously, in the process of using the voice function button on the control device 100 to trigger the input of voice control audio, the user can complete the audio input through the built-in microphone on the control device 100. At this time, the microphone on the control device 100 can convert the user's voice signal into an audio electrical signal, and then send the audio signal to the smart device 200 through the communication connection relationship between the control device 100 and the smart device 200, so that the smart device 200 can obtain the voice control audio input by the user.

[0065] The user can also complete the audio input through the built-in or external audio input device 231 of the smart device 200. That is, after the user presses the voice function button on the control device 100, the smart device 200 can be triggered to start the voice interaction function, and the audio data within a specific time period can be collected through the built-in or external audio input device 231. Therefore, the sound emitted by the user within a specific time period will be input into the smart device 200 as voice control audio, that is, the smart device 200 obtains the voice control audio input by the user.

[0066] In some embodiments, the smart device 200 may support far-field voice input, and the user may trigger the smart device 200 to start the voice interaction function through a specific wake-up word. The wake-up word may be a voice clip containing specific content. The content of the wake-up word may be set by the operator of the smart voice system, for example: "Hi! Xiao×", "Hey! ××", "Xiao× Xiao×", etc. The content of the wake-up word may also support user customization, so that the user can customize the wake-up content according to their own habits.

[0067] When the smart device 200 detects that the user makes a voice with the same content as the wake-up word, the smart device 200 can be triggered to start the voice interaction function, and collect the user's voice within the specific time period of starting the voice interaction function, so as to obtain the voice control audio input by the user. Depending on the user's input habits, voice input can be completed through a single conversation or through multiple conversations. For example, the user can make a voice "Hi! Xiao×, I want to watch ××'s movie". At this time, the smart device 200 can trigger the voice interaction function according to the "Hi! Xiao×" in the voice, and use "I want to watch ××'s movie" as the actual voice interaction content, that is, the voice control audio input by the user.

[0068] For another example, the user can first say "Hi! Xiaox" to trigger the smart device 200 to start the voice interaction function. The smart device 200 can judge the voice content input by the user. When the voice content only contains the wake-up word content and does not contain other substantial content, it can control the audio input device to collect the user's voice within a specific time. At this time, the user enters "I want to watch ××'s movie", and the smart device 200 can obtain the voice control audio with the content "I want to watch ××'s movie".

[0069] In some embodiments, the smart device 200 can also trigger the voice control function through a specific control interface. The user can trigger the smart device 200 to turn on the voice interaction function when controlling the smart device 200 to display a specific control interface or performing an interactive operation on a specific control in a specific interface. For example, in the media search interface, a voice input option may be included. The user can click on the voice input option to trigger the smart device 200 to turn on the voice assistant and collect the user's voice in the current environment within a specific time. At this time, if the user utters the voice "media A", the smart device 200 can obtain the voice control audio with the content "media A".

[0070] It should be noted that the methods for acquiring voice-controlled audio described in the above embodiments are only a few typical examples of many voice audio acquisition methods. The voice input methods that can be set by technical personnel in this field based on specific application environments, specific hardware devices and specific applications can all be used as methods for acquiring user voice input in this application.

[0071] Obviously, the user can trigger the voice interaction function in a variety of scenarios when using the smart device 200. For example, the user can trigger the smart device 200 to start the voice interaction function in one of the above methods when the smart device 200 displays the control homepage interface, media resource recommendation interface, setting interface, playback interface, application interface and other interfaces, so that the smart device 200 can obtain the voice control audio input by the user. In order to reduce the impact of the voice interaction function on some applications, the smart device 200 can also disable some voice interaction functions when running some applications. For example, when the smart device 200 runs some game applications, the far-field voice interaction function can be disabled. At this time, the voice control audio input by the wake-up word will not be obtained by the smart device 200, and the voice control audio input by the voice function button on the control device 100 or the specific user interface can still be obtained by the smart device 200 to trigger the smart device 200 to perform the voice interaction function.

[0072] After acquiring the voice control audio input by the user, the smart device 200 can respond to the voice control audio and generate a feedback statement according to the voice control audio. Since the content of the voice control audio is based on the user's voice input, and the content of the voice input is usually a natural language, it will be affected by different people, different voice types, different periods and other factors, and will be expressed in different forms. Therefore, the voice control audio input by the user cannot be directly recognized by the machine, and specific processing of the natural language is required.

[0073] That is Figure 8As shown, in some embodiments, the smart device 200 can convert the voice control audio into voice control text after acquiring the voice control audio, and then extract the query sentence from the voice control text. For example, when the user inputs the voice "Hi! Xiaox, I want to watch ××'s movie", the smart device 200 can convert the audio data into text data through the built-in voice-to-text application of the smart voice system. That is, " <text>=Hi Xiao×, I want to watch ××'s movie." After converting the voice into text data, the smart device 200 can also remove the wake-up words, modal particles, interjections and other meaningless contents in the text data to obtain the query sentence with the content "I want to watch ××'s movie".

[0074] After the query statement is extracted, the query statement can be input into the speech classification model to obtain the speech classification result. Among them, the speech classification model is a machine learning model, which takes the query statement as input and the classification probability of the statement to the semantic classification label as output. The speech classification model can be obtained in advance by model training, that is, by inputting the training data into the initial model, obtaining the classification probability output by the model, and then comparing it with the label in the training data to determine the loss function, so as to adjust the model parameters of the initial model by back propagation. After a large amount of training data input-output-loss-adjustment parameter process, the initial model can be gradually trained to be a speech classification model that can output accurate classification probability. Therefore, after the query statement is input into the speech classification model, the semantics contained in the natural language can be identified through the calculation of the speech classification model.

[0075] Through the speech classification results output by the speech classification model, the smart device 200 can parse the semantics corresponding to the user's voice control audio, and thus can respond according to the parsed semantics. For example, for the query sentence "I want to watch ××'s movie", the smart device 200 can parse the semantics "media resource retrieval", and the index word of the media resource retrieval is "××". Based on this, the smart device 200 can generate a control instruction according to the parsed semantics, that is, a control instruction for controlling the smart device 200 to perform media resource retrieval, and the index word of the retrieval is "××".

[0076] After the control instruction is generated, the smart device 200 implements voice interaction control by executing the generated control instruction. For example, for a media resource search control instruction with the search term "××", the smart device 200 can search the media resource database with "××" as the keyword by executing the control instruction, and display the search results through the media resource recommendation interface.

[0077] While executing the control instruction, the smart device 200 can also obtain a feedback statement that is compatible with the control instruction. The feedback statement is used to prompt the user of the execution result of the control instruction. For example, for the query statement "I want to watch ××'s movie", it can be determined that the corresponding feedback statement is "A movie of ×× has been found for you". Obviously, the content of the feedback statement is affected by the voice control audio content input by the user, and is also affected by the execution result of the control instruction. For example, when the user inputs "I want to watch YY's movie", but no media asset items related to "YY" are retrieved in the media asset database, the corresponding feedback statement content is obtained as "I did not find YY's movie".

[0078] It should be noted that, in this embodiment, the feedback sentence generated according to the voice control audio is a text-based data, which is used to indicate the content corresponding to the voice broadcast audio. Therefore, in practical applications, the smart device 200 can generate feedback sentences with different contents according to different usage environments, user interfaces and user types. For example, for child users, more interjections and interjections can be included in the feedback sentence to meet the interaction habits of child users.

[0079] Because in the process of intelligent voice interaction, the voice interaction function in some scenarios is only associated with the input voice content. For example, the user can input the voice "Hi! Xiao×, turn off the power" in any interface. At this time, the smart device 200 can directly respond to the input voice content and perform the shutdown action. The voice interaction function in some scenarios is associated with the specific content of the current display interface. For example, when the user inputs the voice "turn the page" in the media resource list interface, it can trigger the smart device 200 to perform a page turning operation on the currently displayed list interface. Therefore, the feedback statement generated for the voice control audio also needs to be adapted according to the content of the current display interface.

[0080] That is, in some embodiments, the smart device 200 also supports a visual question-and-answer function, that is, when the smart device 200 displays a certain content (including images, videos, and UI interfaces), the user can trigger the smart device 200 to enable the visual question-and-answer function by inputting a voice with specific content and format. Such as "What kind of plant is this in the picture", "What TV show did this person play in the TV", etc. When the smart device 200 enables the visual question-and-answer function, it can first perform image recognition on the currently displayed content, and display the image recognition results through the voice broadcast function. For example, "This is five-colored Su. Five-colored Su is an upright or ascending herbaceous plant of the Lamiaceae family and the genus Coleus.", "××'s works include "Media Asset A", "Media Asset B" and "Media Asset C", etc."

[0081] In this regard, in the step of generating a feedback statement according to the voice control audio, the smart device 200 can also obtain the image information of the currently displayed user interface, and input the image information into the image recognition model to obtain the image recognition result. Among them, the image recognition model is also a deep learning model, which takes the image as input and the classification probability of the image for a specific recognition result as output. Similar to the voice classification model, the image recognition model also needs to be trained with a large amount of training data of images with classification labels.

[0082] After the image information is input into the image recognition model, the image recognition result, i.e., the classification probability of the image information for a specific result, can be obtained through calculation by the image recognition model, and then a feedback statement can be generated based on the voice classification result and the image recognition result. For example, when the user inputs the voice "What plant is this" when the smart device 200 displays the playback interface of the photo containing "Five-color Su", the smart device 200 can generate a search instruction after receiving the voice. By executing the search instruction, the smart device 200, on the one hand, takes a screenshot of the current display interface to obtain the image information, and inputs the image information into the image recognition model to identify the five-color Su in the image. On the other hand, the smart device 200 can search according to the image recognition result "Five-color Su" to obtain a description related to the five-color Su. Then, a feedback statement is generated based on the relevant description, i.e., "This is five-color Su. Five-color Su is an upright or ascending herbaceous plant of the Lamiaceae family and the genus Coleus.".

[0083] Since different application scenarios determine feedback statements in different ways, after the step of obtaining the voice classification result, the smart device 200 can first determine the control type to which the voice classification result belongs. Among them, the control types include visual question and answer types and non-visual question and answer types. The smart device 200 can be determined based on the current user interface type and the voice control audio content input by the user. For example, when the current user interface type is a playback interface or contains a picture display window, and the voice control audio input by the user contains referential words such as "this is", "this", "currently", "on TV", and "in the picture", it can be determined that the current control type is a visual question and answer type.

[0084] According to the control type to which the speech classification result belongs, if the speech classification result belongs to the visual question and answer category, the image information of the currently displayed user interface is obtained, and image recognition is performed in the manner of the above embodiment so as to generate a feedback statement based on the speech classification result and the image recognition result. If the speech classification result belongs to the non-visual question and answer category, that is, the current speech interaction process is irrelevant to the currently displayed content, and therefore the feedback statement can be generated only based on the speech classification result.

[0085] After generating the feedback sentence, the smart device 200 can perform voice broadcast according to the feedback sentence, that is, generate voice broadcast audio according to the feedback sentence, and play it through the audio output device 271. In order to obtain a better voice broadcast effect, in some embodiments of the present application, the smart device 200 can give an emotional effect to the voice broadcast process, that is, under different emotional types, use different intonations, speech speeds, and timbres to play the feedback sentence.

[0086] In some embodiments, the smart device 200 can determine the emotion type of the voice broadcast according to the current scene classification. That is, the smart device 200 can detect the current scene classification after generating the feedback statement. Among them, the scene classification can include usage scenarios and user intention scenarios. The usage scenario can be determined by the currently displayed user interface type, interface logic, application type and display content. For example, when a user opens a reading application, the smart device 200 will display the reading application interface at runtime. At this time, if the user triggers the voice interaction function, the usage scenario can be determined as a reading scenario based on the current application type. The user intention scenario needs to be determined by semantically analyzing the voice control audio input by the user, that is, the smart device 200 can input the voice control audio input by the user into the scene classification model, and obtain the scene classification to which the user query statement in the voice control audio belongs according to the scene classification result output by the scene classification model.

[0087] For user intention scenarios, multiple scenario categories can be preset according to the actual use of the smart device 200. For example, food, calories, animals, plants, cars, attractions, film and television knowledge questions and answers, general simple questions and answers, etc. Each scenario category is set with a corresponding query statement form, as shown in the following table:

[0088]

[0089] For the detected scene classification, the emotion type adapted to each scene classification can be preset to establish an emotion classification table. After the scene classification is detected, the smart device 200 calls the emotion classification table. Since the emotion classification table includes a mapping relationship between the preset scene classification and the default emotion, the smart device 200 can use the scene classification to match the default emotion in the emotion classification table to play the feedback sentence according to the default emotion.

[0090] Among them, the emotions corresponding to the emotion categories are mainly affected by factors such as intonation, speaking speed, and timbre. Therefore, when determining the default emotions, different intonations can be selected according to different scenarios, and specific timbres can be selected according to different scenarios. For example, in the emotion classification table, the default food visual question and answer scenario can be reported using a slow speaking speed, a steady tone, and a middle-aged male timbre. The calorie visual question and answer scenario uses a fast speaking speed, a high-pitched tone, and a fitness anchor timbre for voice broadcasting.

[0091] Obviously, in order to cater to the preferences of different users, the intelligent voice system of the smart device 200 can also support customized intonation, speech speed and timbre. In addition, users can choose other favorite intonations, speech speeds and timbre for each scene. For example, in terms of timbre, users can set timbre options such as "sweet little girl", "intellectual sister", "magnetic man" through the voice assistant setting interface. In addition, users can also record their own voices or other favorite timbres. In order to maintain the anthropomorphic experience, one timbre is configured, and multiple intonations can be configured according to different scenes.

[0092] Since in actual conversations, the voice changes due to emotions are diverse, and the final determination of the emotion type is also affected by multiple factors, in some embodiments of the present application, the smart device 200 can also calculate the emotion score of the feedback sentence after generating the feedback sentence. The emotion score can be affected by multimodal information of one or more combinations of the scene classification corresponding to the feedback sentence, the content of the feedback sentence, and the image recognition result.

[0093] For example, when the output is an image description sentence or a regular voice answer sentence, the user's inquiry sentence and the image description sentence can be emotionally scored to output different speech synthesis broadcast results according to different emotions, that is, emotional types such as happy, sad, calm, etc. Since emotional types such as disgust, anger, and surprise are not suitable for answering voice, they are not considered in the voice broadcast scenario of the smart device 200.

[0094] Different emotion types correspond to different score values. For example, happiness is scored as +2, sadness is scored as -2, and calmness is scored as 0. If any item in the multimodal information identifies a corresponding emotion, the score corresponding to the emotion is accumulated. Then the scores corresponding to multiple modal information are summed up to obtain the cumulative score of the multimodal information as the emotional score of the feedback sentence.

[0095] For example, the sentence input by the user is "What's on TV?", which is the default calm emotion, that is, the scene classification score is 0. However, through image recognition, it is determined that there are cows eating grass in the currently displayed picture, and the feedback sentence "A group of cute cows are eating grass" is generated. The sentence describes a happy emotion. At this time, the multimodal information cumulative score can be calculated as 0+(+2)=2, so the happy emotion can be automatically selected to play the feedback sentence "A group of cute cows are eating grass".

[0096] In some embodiments, the sentiment score is a weighted summation result of the multimodal information classification scores of one or more combinations of the scene classification corresponding to the feedback sentence, the content of the feedback sentence, and the image recognition result. Among them, the weights used in the weighted summation operation can be manually set by the user, or can be converted and calculated according to any item in the multimodal information. That is, when calculating the final sentiment score, the smart device 200 can also perform a comprehensive calculation according to the positive or negative answer in the feedback sentence after calculating the cumulative score of the multimodal information, that is, the weight for the positive answer is 1, and the weight for the negative answer is 0, so as to calculate the product of the weight corresponding to the positive answer or negative answer in the feedback sentence and the cumulative score of the multimodal information, so as to obtain the final score.

[0097] For example, if the sentence input by the user is "Is the person on TV crying?", the speech classification model can identify that the current semantics contains sad emotions, that is, the scene classification score is -2, and the image recognition model determines that the current display of the smart device 200 is a picture of a person crying, that is, the image recognition result score is -2, and the multimodal information accumulation score is -4. Combined with the content of the feedback sentence "yes", it is a positive answer. The final emotion score is the product of the multimodal information accumulation score and the positive answer, that is, -4×1=-4. At this time, the smart device 200 can determine that the current "sad" emotion type has the highest score, so the sad emotion can be selected to play "yes".

[0098] Similarly, if the sentence input by the user is "Is the person on TV crying?", that is, the scene classification score is -2, and image recognition determines that the person's expression in the currently displayed picture is calm, that is, the image recognition result score is 0, then a negative answer feedback sentence "no" will be generated. Therefore, the final score is -2×0=0, so the default calm emotion is selected to play the feedback sentence "no".

[0099] After calculating the emotional score of the feedback sentence, the target emotional type can be determined according to the emotional score, and the feedback sentence can be played according to the target emotional type. Each emotional type corresponds to a scoring interval, and when it is determined that the emotional score of the current feedback sentence is in the corresponding interval, the target emotional type is determined. For example, feedback sentences with an emotional score greater than 0 can be set to be played with a happy emotion, feedback sentences with an emotional score less than 0 can be set to be played with a sad emotion, and feedback sentences with an emotional score equal to 0 can be played with a calm emotion. In order to obtain a richer and more diverse voice broadcast effect, in some embodiments of the present application, more emotional types can be further set according to the actual conversation tone and emotional rules.

[0100] It can be seen from the above technical solution that in the above embodiment, the smart device 200 can use multimodal information to comprehensively evaluate the current language environment, and adopt a specific emotion scoring mechanism to more accurately determine the emotion type corresponding to the current context, so that the voice broadcast process is more in line with the user's current context and improves the user experience.

[0101] In some embodiments of the present application, the smart device 200 can also determine the sentiment score through a sentiment classification model. Among them, the sentiment classification model can identify the sentiment type of each item in the multimodal information, so as to obtain the classification probability of each information corresponding to a specific sentiment type. Since the specific expression form of each item in the multimodal information is different, for example, the scene classification result and the feedback statement content are in text form, and the image recognition result is in image form, different sentiment classification models can be established when performing classification operations. For example, a first sentiment classification model with feedback statements as input; a second sentiment classification model with image recognition results as input; a scene classification model with inquiry statements as input. Obviously, the inputs of the first sentiment classification model, the second sentiment classification model, and the scene classification model are all the classification probabilities of the input data for a specific sentiment type, that is, the sentiment scores.

[0102] Therefore, if Fig. 9 As shown, in the step of calculating the sentiment score of the feedback statement, the smart device 200 can input the feedback statement into the first sentiment classification model, the image recognition result into the second sentiment classification model, and the inquiry statement into the scene classification model. Then, the classification results of the three model outputs are obtained respectively, so as to obtain the first sentiment score, the second sentiment score, and the third sentiment score. Among them, in order to improve the applicability of the model, the model output result of the scene classification model can be the classification probability of the input data for a certain scene classification. For this purpose, the smart device 200 can also search for the sentiment score under the corresponding scene according to the mapping relationship comparison table between the preset sentiment score and the scene classification, that is, the third sentiment score. Finally, the first sentiment score, the second sentiment score, and the third sentiment score are weighted and summed, and the weighted sum result is used as the sentiment score of the feedback statement.

[0103] It should be noted that, since the smart device 200 can select one or more items of multimodal information for sentiment scoring during the voice interaction process, in some embodiments, the smart device 200 can only input the corresponding sentiment classification model for some items in the feedback statement, the inquiry statement, and the image recognition result to calculate, for example, the feedback statement is input into the first sentiment classification model to obtain the first sentiment score output by the first sentiment classification model. At the same time, the image recognition result is input into the second sentiment classification model to obtain the second sentiment score output by the second sentiment classification model. Then the sentiment score is calculated based on the first sentiment score and the second sentiment score.

[0104] It can be seen that in the above embodiment, the smart device 200 can perform a weighted calculation based on the classification results corresponding to the multimedia information to determine the emotional score of the feedback statement. Different modal items can set corresponding weights according to specific application scenarios. For example, for the visual question and answer scenario, since the currently displayed image content has a greater impact on the voice interaction function, the second emotional score corresponding to the image recognition result can be set to have a larger weight to meet the basic needs of the visual question and answer scenario.

[0105] After calculating and obtaining the emotion score in the manner provided in the above embodiment, the smart device 200 can determine the target emotion type according to the emotion score, and play the feedback sentence according to the target emotion type. In the process of playing the feedback sentence, the smart device 200 can also adopt different playing methods according to the recording situation of the voice segment in the current intelligent voice system.

[0106] That is, in some embodiments, if the current intelligent voice system has pre-recorded voice clips of each emotion type (at least including the current target emotion type), the smart device 200 can first call the voice database of the target emotion type, and match the target audio data corresponding to the feedback sentence in the voice database, and output the voice broadcast audio by playing the target audio data.

[0107] For example, if the emotional score mechanism determines that the emotional score of the sad emotional type is the highest in the context corresponding to the current feedback sentence, the smart device 200 can determine that the target emotional type is "sad". Therefore, the smart device 200 can call a voice database of the sad type, that is, a voice package, and match the voice segment with the same word content as the feedback sentence in the voice database. Finally, by splicing the multiple matching voice segments, a voice broadcast audio containing a sad emotional tone is formed, and played through the audio output device 271.

[0108] like Fig.10 As shown, in some embodiments, the smart device 200 can also realize the generation of voice broadcast audio based on text to speech (Text To Speech, TTS) technology. That is, a TTS model can be built-in in the smart device 200, and the TTS model is also a neural network model, including an emotion embedding network (Emotion Embedding Network, EEN), an auxiliary network (auxil iary network) and a speech synthesis framework network (tacotron2). When synthesizing the voice broadcast audio, the feedback sentence is input into the speech synthesis framework network as a feature sequence (character sequence), and the speech synthesis framework network is encoded (encoder), attention mechanism (attent ion), decoding (decoder) and spectrum processing (melspectrogram) and other processing processes, and finally processed by a neural vocoder (neural vocoder), and the output is an audio containing emotional color for playback.

[0109] For the process of generating speech broadcast audio by the TTS model, the emotion embedding network can embed emotion factors in the encoding according to the determined target emotion type during encoding, so that the final output audio can have the voice, speaking speed and intonation of the target emotion type. In addition, during the spectrum processing process, the auxiliary network can also be used to classify the spectrum processing results to output the results in the form of gram matrices, and perform type loss calculation with the target emotion type, so as to adjust the network parameters of the emotion embedding network through back propagation, so that the emotion embedding network can be continuously trained to improve the classification accuracy.

[0110] In addition, the smart device 200 can also add a speaker embedding network to the emotion embedding network based on the emotion-based TTS model to form an emotion disentangling module (EDM), thereby increasing the emotion and timbre effects during the encoding process of the speech synthesis framework network. Similarly, in the spectrum processing process, by adding a timbre-related auxiliary network, the auxiliary network can output timbre-related results for type loss calculation. The specific application model can use a self-developed TTS model or a related TTS product, which will not be repeated here.

[0111] In order to adapt to different users, in some embodiments, the intelligent voice interaction method also supports voice broadcasting in multiple languages. That is, the smart device 200 can select the main language according to the application area, such as Chinese mode, English mode, etc. In the main language mode, the user can ask questions using the language type corresponding to the main language, and the smart device 200 will use the voice corresponding to the main language to broadcast after determining the feedback sentence. It can also be expanded to other language types such as German, French, Spanish, Japanese, etc. to meet different user needs.

[0112] For the TTS process of multiple language types, the TTS model built into the smart device 200 may include an encoder / decoder module, a speaker network, a language network, and a neural vocoder. Among them, the encoder / decoder module can synthesize the feedback text "sequence to sequence" and is a transformer-based encoder-decoder structure that can directly predict the mel spectrogram from the text. The pronunciation network and the language network can be used to control the synthesis of speech of different speakers and languages. The pronunciation network can be a network that can distinguish speaker information. For example, the pronunciation network can be a mapping table (lookup table, LUT) network, which uses the mapping table network to make common features have similar dynamic ranges, thereby mapping text to phonemes.

[0113] Among them, phonemes can be represented by one-hot vectors through LUT in the encoding module. In order to avoid confusion caused by similar pronunciations in different languages, the phonemes of each language are independent. The phoneme sets of various languages ​​are also independent of each other, that is, each language has a separate phoneme table. Similarly, the language network has a similar structure to the pronunciation network, and the language itself can also be represented by a one-hot vector. The neural vocoder is a neural network decoder that can convert mel spectrograms into audio. Therefore, related neural network decoders such as WaveNet, Griffin-Lim, etc. can be directly used.

[0114] Based on the above-mentioned voice interaction form supporting multiple languages, in some embodiments of the present application, the smart device 200 can also call a multilingual speech synthesis TTS model after obtaining the voice control audio input by the user, and input the voice control audio into the TTS model to identify the language type to which the voice control audio belongs. And when playing the feedback sentence according to the target emotion type, the TTS model is used to synthesize the output waveform of the feedback sentence, so as to play the feedback sentence according to the output waveform, thereby realizing the multilingual interaction function.

[0115] In some embodiments, in order to support multi-language voice interaction, a TTS model can also be obtained in advance by model training, that is, the smart device 200 can obtain a training data set before calling the multi-language speech synthesis TTS model, and the training data set includes training data in multiple languages. Since there is an imbalance in the amount of data in various languages, a corresponding data sampling strategy can be set in the model training stage. That is, the smart device 200 can set the sampling probability of each language by traversing the number of training samples corresponding to each language in the training data set, and then according to the number of training samples corresponding to each language and the preset sampling parameters.

[0116] For example, for the i-th language, the number of utterances is Ni. Then in the entire training set, the probability of sampling from the i-th language is:

[0117]

[0118] Where Ci is the probability of sampling in the i-th language; Ni is the number of utterances in the i-th language; ∑ j N j is the total number of utterances in the language set.

[0119] To sample each batch of data, the probability needs to add a sampling parameter α, where α∈[0,1]. At this time, the probability of each language is pi, then:

[0120]

[0121] In the formula, pi is the distribution probability of the i-th language; Ci is the sampling probability of the i-th language; j is the total number of languages; α is the preset sampling parameter. It can be seen that when α = 0, the language distribution probability is uniformly distributed; and when α = 1, the model receives the real data distribution. To alleviate the problem of data imbalance, the sampling parameter α can be set to 0.2.

[0122] In addition, since there may be many speakers of the same language, there may be serious data imbalance among these speakers. Therefore, the above data balancing strategy can be used to sample the training data of different speakers of each language, that is, to sample training data in the training data set according to the sampling probability, and input the sampled training data into the TTS model to train the TTS model.

[0123] Based on the intelligent voice interaction method provided in the above embodiment, a smart device 200 is also provided in some embodiments of the present application. The smart device 200 includes: a display 260, an audio input device 231, an audio output device 271 and a controller 250. The display 260 is configured to display a user interface; the audio input device 231 is configured to receive voice control audio input by the user; the audio output device 271 is configured to output voice broadcast audio; and the controller 250 is configured to execute the following program steps:

[0124] Get the voice control audio input by the user;

[0125] In response to the voice control audio, generating a feedback sentence based on the voice control audio;

[0126] Calculating the sentiment score of the feedback sentence, the sentiment score being a weighted sum of classification scores of multimodal information corresponding to the feedback sentence; the multimodal information comprising a combination of one or more of scene classification, feedback sentence content, and image recognition results;

[0127] A target emotion type is determined according to the emotion score, and the feedback sentence is played according to the target emotion type.

[0128] It can be seen from the above technical solution that the smart device 200 provided in the above embodiment can generate a feedback sentence according to the voice control audio after the user inputs the voice control audio, and calculate the emotional score of the feedback sentence according to the set emotional scoring mechanism, thereby determining the target emotional type according to the emotional score, and finally playing the feedback sentence according to the target emotional type to form a voice broadcast. The smart device 200 can perform a weighted sum calculation on the classification scores of multimodal information such as scene classification, feedback sentence content, and image recognition results through the feedback sentence, obtain the emotional score, and finally determine the target emotional type, so that the voice broadcast audio can be played with a more reasonable emotional type, alleviating the problem of poor voice broadcast effect.

[0129] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.< / text>

Claims

1. A smart device, characterized in that: include: a display configured to display a user interface; An audio input device configured to receive voice control audio input by a user; An audio output device configured to output voice broadcast audio; The controller is configured as: Get the voice control audio input by the user; In response to the voice control audio, parsing the semantics of the voice control audio and generating a control instruction according to the semantics, and generating a feedback statement according to the voice control audio, wherein the feedback statement is used to prompt a user of the execution result of the control instruction; Calculating the sentiment score of the feedback sentence, the sentiment score being a weighted sum of the multimodal information classification scores corresponding to the feedback sentence; the multimodal information comprising a combination of one or more of a scene classification, a feedback sentence content, and an image recognition result, wherein the scene classification comprises a usage scene and a user intention scene, the usage scene being determined according to the currently displayed user interface type, interface logic, application type, and display content, and the user intention scene being determined after semantic analysis of the voice control audio input by the user; A target emotion type is determined according to the emotion score, and the feedback sentence is played according to the target emotion type, so that the playing of the feedback sentence has a corresponding tone, speech speed and timbre.

2. The smart device according to claim 1, characterized in that: The controller is further configured to: In the step of generating a feedback sentence according to the voice control audio, converting the voice control audio into a voice control text; Extracting a query sentence from the voice control text; Inputting the query sentence into a speech classification model to obtain a speech classification result; Get the image information of the currently displayed user interface; Inputting the image information into an image recognition model to obtain an image recognition result; A feedback sentence is generated according to the speech classification result and the image recognition result.

3. The smart device according to claim 2, characterized in that: The controller is further configured to: In the step of calculating the sentiment score of the feedback sentence, the feedback sentence is input into a first sentiment classification model to obtain a first sentiment score output by the first sentiment classification model; Inputting the image recognition result into a second sentiment classification model to obtain a second sentiment score output by the second sentiment classification model; The sentiment score is calculated based on the first sentiment score and the second sentiment score.

4. The smart device according to claim 3, characterized in that: The controller is further configured to: In the step of calculating the sentiment score according to the first sentiment score and the second sentiment score, the query statement is input into a scene classification model; Obtaining a scene classification result output by the scene classification model for the query statement; Calculating a third emotion score of the scene classification result; The sentiment score is calculated, where the sentiment score is a weighted sum of the first sentiment score, the second sentiment score, and the third sentiment score.

5. The smart device according to claim 2, characterized in that: The controller is further configured to: After the step of obtaining the speech classification result, determining the control type to which the speech classification result belongs, the control type including visual question answering type and non-visual question answering type; If the speech classification result belongs to the visual question answering category, obtaining image information of the currently displayed user interface; If the speech classification result belongs to the non-visual question answering category, a feedback sentence is generated according to the speech classification result.

6. The smart device according to claim 1, characterized in that: The controller is further configured to: Before the step of calculating the sentiment score of the feedback sentence, obtaining the scene classification to which the user inquiry sentence in the voice control audio belongs; Calling an emotion classification table, wherein the emotion classification table includes a mapping relationship between preset scene classifications and default emotions; The default emotion is matched in the emotion classification table using the scene classification to play the feedback sentence according to the default emotion.

7. The smart device according to claim 1, characterized in that: The controller is further configured to: In the step of playing the feedback sentence according to the target emotion type, calling a speech database of the target emotion type; Matching target audio data corresponding to the feedback sentence in the voice database; The target audio data is played to output voice broadcast audio.

8. The smart device according to claim 1, characterized in that: The controller is further configured to: After obtaining the voice control audio input by the user, call the multi-language speech synthesis TTS model; Inputting the voice control audio into the TTS model to identify the language type to which the voice control audio belongs; In the step of playing the feedback sentence according to the target emotion type, synthesizing the output waveform of the feedback sentence using the TTS model; The feedback sentence is played according to the output waveform.

9. The smart device according to claim 8, characterized in that: The controller is further configured to: Before the step of calling the multilingual speech synthesis TTS model, a training data set is obtained, wherein the training data set includes training data in multiple languages; Traversing the number of training samples corresponding to each language in the training data set; According to the number of training samples corresponding to each language and the preset sampling parameters, set the sampling probability of each language; Sampling training data from the training data set according to the sampling probability; The sampled training data is input into the TTS model to train the TTS model.

10. An intelligent voice interaction method, characterized in that: Applied to smart devices, the intelligent voice interaction method includes: Get the voice control audio input by the user; In response to the voice control audio, parsing the semantics of the voice control audio and generating a control instruction according to the semantics, and generating a feedback statement according to the voice control audio, wherein the feedback statement is used to prompt a user of the execution result of the control instruction; Calculating the sentiment score of the feedback sentence, the sentiment score being a weighted sum of the multimodal information classification scores corresponding to the feedback sentence; the multimodal information comprising a combination of one or more of a scene classification, a feedback sentence content, and an image recognition result, wherein the scene classification comprises a usage scene and a user intention scene, the usage scene being determined according to the currently displayed user interface type, interface logic, application type, and display content, and the user intention scene being determined after semantic analysis of the voice control audio input by the user; A target emotion type is determined according to the emotion score, and the feedback sentence is played according to the target emotion type, so that the playing of the feedback sentence has a corresponding tone, speech speed and timbre.

Images