A server, an intelligent device and an intelligent voice control method

Abstract

The present application provides a server, an intelligent device, and an intelligent voice control method. After a user inputs a voice control instruction, the method can first send the voice control instruction to the server. The server extracts user intention information from the voice control instruction and matches intelligent devices that can implement the user intention information in a mapping database to generate a target device list. Then, it finds the execution device closest to the user or with the highest execution weight in the target device list and sends the voice control instruction to the execution device to respond to the voice control instruction input by the user. The method can determine the execution device for responding to the user's voice control instruction based on the user intention matching result and the sound source localization result. Therefore, when there are multiple intelligent devices capable of responding to voice control instructions in a smart home system, only the execution device makes a response, alleviating the problems of control chaos and the situation where only the execution device responds but the feedback cannot support the control service.

Description

Technical Field

[0001] This application relates to the technical field of smart home, and in particular, to a server, a smart device, and a smart voice control method. Background Art

[0002] Smart voice control is a new type of interaction method. It can perform semantic recognition on the voice information input by the user, and then control the operation of the device according to the semantic recognition result. In order to implement the interaction process based on smart voice control, a smart voice system can be built into the smart device. The smart voice system can be composed of a hardware part and a software part. Among them, the hardware part mainly includes a microphone, a speaker, and a controller, which are used to receive, feedback, and process voice information; the software part mainly includes a voice conversion module, a natural language processing module, and a control module, which are used to convert the input sound signal into a text signal and form specific control instructions for control.

[0003] When the user uses the smart voice system, the smart device can detect the specific voice input by the user through the hardware part, and then call the software part to process the input voice, convert the voice into a control instruction, and finally execute the corresponding control in response to the control instruction, and feedback the execution result through the hardware part. For example, a smart voice system can be built into a smart TV. When the user inputs the voice "I want to watch a movie" through the microphone, the smart TV can be triggered to play movie media for the user to watch.

[0004] Smart voice control can also be applied to a smart home system composed of multiple smart devices. For example, the smart home system can include smart devices such as a smart TV, a smart speaker, and a smart refrigerator that establish communication connection relationships with each other. Since multiple smart devices in the smart home system can all be built with smart voice systems, multiple smart devices will respond to the voice input by the user at the same time, resulting in control chaos. For example, when the user inputs the voice "I want to watch a movie", both the smart TV and the smart speaker will respond and respectively feedback different execution results, that is, the smart TV can jump to the movie playing interface, while the smart speaker will feedback error prompt information such as "I can't complete", affecting the user's operation experience. Summary of the Invention

[0005] This application provides a server, a smart device, and a smart voice control method to solve the problem of control chaos caused by multiple smart devices responding to the user's voice at the same time.

[0006] In a first aspect, the present application provides a server, including: a storage module, a communication module, and a control module. Among them, the storage module is configured to store a mapping database, and the mapping database includes the mapping relationship between user intention information and device capability information; the communication module is configured to establish communication connections with multiple intelligent devices; the control module is configured to execute the following program steps:

[0007] Receive a voice control instruction sent by an intelligent device;

[0008] In response to the voice control instruction, extract user intention information from the voice control instruction;

[0009] Generate a target device list, and the target device list includes intelligent devices that can implement the user intention information obtained by matching in the mapping database;

[0010] Search for an execution device in the target device list, and the execution device is the intelligent device closest to the location where the voice control instruction is initiated;

[0011] Send the voice control instruction to the execution device to trigger the execution device to perform a voice response for the voice control instruction.

[0012] In a second aspect, the present application further provides an intelligent device, including: an audio input device, an audio output device, a communicator, and a controller. Among them, the audio input device is configured to detect voice audio data input by a user; the audio output device is configured to play a voice response; the communicator is configured to establish a communication connection with a server; the controller is configured to execute the following program steps:

[0013] Obtain voice audio data input by a user;

[0014] Generate a voice control instruction according to the voice audio data;

[0015] Send the voice control instruction to the server, so that the server extracts user intention information from the voice control instruction, and matches intelligent devices that can implement the user intention information in the mapping database to generate a target device list; and search for an execution device in the target device list and send the voice control instruction;

[0016] Perform a voice response for the voice control instruction.

[0017] In a third aspect, the present application further provides an intelligent voice control method, which is applied to a smart home system. The smart home system includes a server and intelligent devices, and a communication connection is established between the server and the intelligent devices; the intelligent voice control method includes the following steps:

[0018] The intelligent device obtains the voice audio data input by the user and generates a voice control instruction according to the voice audio data;

[0019] The intelligent device sends the voice control instruction to the server;

[0020] The server extracts the user intention information from the voice control instruction and matches the intelligent devices capable of implementing the user intention information in the mapping database to generate a target device list;

[0021] The server searches for an execution device in the target device list and sends the voice control instruction to the execution device;

[0022] The intelligent device serving as the execution device performs a voice response for the voice control instruction.

[0023] As can be seen from the above technical solutions, the present application provides a server, an intelligent device, and an intelligent voice control method. After the user inputs a voice control instruction, the method can first send the voice control instruction to the server. The server extracts the user intention information from the voice control instruction and matches the intelligent devices capable of implementing the user intention information in the mapping database to generate a target device list. Then, it searches for the execution device closest to the user or with the highest execution weight in the target device list and sends the voice control instruction to the execution device to respond to the voice control instruction input by the user. The method can determine the execution device for responding to the user's voice control instruction based on the user intention matching result and the sound source localization result. Therefore, when there are multiple intelligent devices capable of responding to voice control instructions in the smart home system, only the execution device makes a response, alleviating the problems of control chaos and only the execution device responding but the feedback being unable to support the control service. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0025] Figure 1 is the usage scenario of the smart home system in the embodiment of the present application;

[0026] Figure 2 is the hardware configuration diagram of the intelligent device in the embodiment of the present application;

[0027] Figure 3 is the schematic diagram of the voice interaction process in the embodiment of the present application;

[0028] Figure 4Schematic diagram of the voice interaction effects of multiple intelligent devices in the embodiments of the present application;

[0029] Figure 5 Schematic flowchart of the intelligent voice control method on the intelligent device side in the embodiments of the present application;

[0030] Figure 6 Schematic flowchart of the intelligent voice control method on the server side in the embodiments of the present application;

[0031] Figure 7 Schematic flowchart of the process for finding an execution device in the embodiments of the present application;

[0032] Figure 8 Schematic flowchart of the process for an intelligent device to go online in the embodiments of the present application;

[0033] Figure 9 Sequence diagram of the intelligent voice control method on the server side in the embodiments of the present application;

[0034] Figure 10 Sequence diagram of the intelligent voice control method on the intelligent device side in the embodiments of the present application. Detailed implementation manners

[0035] The embodiments will be described in detail below, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following embodiments do not represent all implementation manners consistent with the present application. They are only examples of the systems and methods consistent with some aspects of the present application detailed in the claims.

[0036] The smart home system in the embodiments of the present application is a network system based on a specific area network and established based on unified control services. The smart home system may include multiple intelligent devices 200 that establish communication connection relationships with each other. The multiple intelligent devices 200 can achieve the communication connection relationship between devices by accessing the same local area network. The multiple intelligent devices 200 can also directly form a peer-to-peer network through a unified communication protocol to achieve communication connection. For example, the multiple intelligent devices 200 can communicate with each other by connecting to the same wireless local area network. Also for example, an intelligent device 200 can also establish a communication connection with other multiple intelligent devices 200 through means such as Bluetooth, infrared, cellular network, and power line carrier communication.

[0037] Among them, the intelligent device 200 refers to a device with communication functions that can receive, send, execute control instructions, and implement specific functions. The intelligent device 200 includes, but is not limited to, intelligent display devices, intelligent terminals, intelligent home appliances, intelligent gateways, intelligent lighting devices, intelligent audio devices, game devices, etc. Multiple intelligent devices 200 that make up the smart home system can be of the same type or different types. For example, as Figure 1 shown, in the same smart home system, it can include smart TVs, smart speakers, smart refrigerators, multiple smart lamps, etc. These intelligent devices 200 can be distributed at different locations to meet the usage requirements at the corresponding locations.

[0038] It should be noted that the smart home system described in this application does not limit the application scope of the solution to be protected in this application. That is, in actual applications, the server, intelligent device, and intelligent voice control method provided in this application are not limited to being applied in the smart home field, and are also equally applicable to other systems that support intelligent voice control, such as intelligent office systems, intelligent service systems, intelligent management systems, industrial production systems, etc.

[0039] According to the actual functions of the intelligent device 200, the intelligent device 200 has a specific hardware configuration. As Figure 2 shown, taking a display device as an example, the intelligent device 200 with a display function may include at least one of a tuner demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0040] In some embodiments, the controller 250 includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to n interfaces for input / output.

[0041] In some embodiments, the display 260 includes a display screen component for presenting a picture, and a driving component for driving image display, a component for receiving an image signal output from the controller, and for displaying video content, image content, and a menu control interface, as well as a user control UI interface, etc.

[0042] In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0043] In some embodiments, the tuner demodulator 210 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as and EPG data signals.

[0044] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more of the following: High-Definition Multimedia Interface (HDMI), analog or digital high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by the above-mentioned multiple interfaces.

[0045] In some embodiments, the controller 250 controls the operation of the smart device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the smart device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 may perform operations related to the object selected by the user command.

[0046] In some embodiments, the user may input a user command on the graphical user interface (GUI) displayed on the display 260, and then the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and then the user input interface receives the user input command by identifying the sound or gesture through the sensor.

[0047] In some embodiments, the smart device 200 also performs data communication with the server 400. The smart device 200 is allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the smart device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of server units.

[0048] In some embodiments, the smart device 200 may be built with an intelligent voice system to support the user's intelligent voice control. The intelligent voice control refers to the interaction process in which the user operates the smart device 200 by inputting voice audio data. To implement the intelligent voice control, the smart device 200 may include an audio input device and an audio output device. Among them, the audio input device is used to collect the voice audio data input by the user, and may be a microphone device built in or externally connected to the smart device 200. The audio output device is used to emit sound to play a voice response. For example, as Figure 3 shown, when the user inputs a wake-up word such as "Hi! Xiao ×" through the audio input device, the smart device 200 may play a voice response of "I'm here" through the audio output device to guide the user to complete subsequent voice input.

[0049] In some embodiments, the intelligent voice system built into the intelligent device 200 also supports the one-shot mode, that is, it supports the "one-shot" mode. In this mode, users can directly achieve control functions through fewer voice inputs. For example, in the traditional mode, if a user wants to control the intelligent device 200 to play movie resources, they need to first input the voice "Hi, Xiao ×". After the intelligent device 200 responds with "I'm here", then input "I want to watch a movie", and then the intelligent device 200 responds with "The following movies are found for you". In the "one-shot" mode, the user can directly input "Hi! Xiao ×, I want to watch a movie", and then the intelligent device 200 directly responds with "The following movies are found for you" after receiving the voice command, reducing the number of voice interactions and improving the efficiency of voice interaction.

[0050] For multiple intelligent devices 200 in the same smart home system, users can control the linkage of multiple devices through intelligent voice. For example, the user can input the voice command "Turn on the bedroom light" through the smart speaker. Then the smart speaker can generate a control command for turning on the light in response to the voice command, and then send the control command to the lamp named "bedroom" in the smart home system to control the turning on of the bedroom light. At the same time, the smart speaker also responds to the user's voice input, that is, plays feedback voice content such as "The bedroom light has been turned on for you".

[0051] When controlling the linkage between multiple intelligent devices 200, the control command can be directly transmitted from the intelligent device 200 that receives the user's voice audio data to the controlled device, or can be transmitted from the intelligent device 200 to a specific relay device such as a router, and then transmitted from the relay device to the controlled device. In some embodiments, the control command can also be transmitted to the controlled device through the server 400. For example, when the user controls a certain intelligent device 200 in the smart home system through the intelligent terminal 300 outside the local area network where the smart home is located, the intelligent terminal 300 can first send the control command to the server 400, and then the server 400 transmits the control command to the intelligent device 200 for control.

[0052] To control the intelligent device 200 in the smart home system, the server 400 can send control instructions and relevant data to any intelligent device 200 individually. For example, for a display device, the user can control the display device through interactive operations to request online playback of media assets, and then the server 400 can feedback media asset data to the display device according to the playback request. For the linked control of multiple intelligent devices 200, the server 400 can send control instructions and relevant data to the smart home system uniformly. For example, when the user uses the smart speaker to control the bedroom lights to turn on, the smart speaker can send the control instructions input by the user to the server 400, and the server 400 then sends feedback data to the smart home system, so that the smart home system sends an on instruction to the bedroom lights and at the same time feedbacks a control response to the smart speaker.

[0053] Some intelligent devices 200 in the smart home system can be built with a complete intelligent voice system. Such intelligent devices 200 can be used as the main control devices, capable of receiving, processing, and responding independently, and at the same time can send control instructions corresponding to the voice audio to other intelligent devices 200. For example, intelligent devices 200 such as display devices, smart speakers, and smart refrigerators can be built with a complete intelligent voice system to receive the voice audio input by the user. Some intelligent devices 200 in the smart home system may not be built with a complete intelligent voice system and only act as controlled devices to receive the control instructions sent by the main control device. For example, intelligent devices such as lights and small household appliances can receive the control instructions transmitted by the display device acting as the main control device to start, stop running, or change the operating parameters.

[0054] As the degree of device intelligence continues to increase, the types of intelligent devices 200 that can be built with a complete intelligent voice system are gradually increasing, so that in the same smart home system, there can be multiple intelligent devices 200 that can be used as the main control devices. At this time, a voice control instruction input by the user may be responded to by multiple intelligent devices 200, which not only wastes computing resources but also causes chaos in the control process.

[0055] For example, when both a smart TV and a smart speaker with a complete intelligent voice system are connected to the smart home system at the same time, when the user inputs the voice "I want to watch a movie", both the smart TV and the smart speaker can receive the voice content and make voice interaction responses respectively, that is, Figure 4As shown, the smart TV can feedback "The following movies are found for you" and display a movie list for the user to select. Since the smart speaker does not support movie playback, the smart speaker can feedback "I don't understand what you're saying" or "The TV has been turned on for you" to control the smart TV to start running. At this time, the smart TV and the smart speaker will feedback voice responses simultaneously, causing confusion in the information heard by the user. And since the smart TV is already in the on state and has responded to the input voice, the process of the smart speaker controlling the startup of the smart TV will not respond and is prone to incorrect control, unable to meet the user's interaction needs.

[0056] It should be noted that the input methods of voice commands can include various forms such as remote control input and far-field voice input. Among them, remote control input means that the user presses the voice button on the remote control supporting the smart device 200 and inputs voice through the microphone on the remote control. At this time, since the remote control used by the user is usually a supporting device for the specific smart device 200, the supporting smart device 200 can directly respond to the voice command input by the user. For example, when the user wants to control the smart TV by voice, the user can input a voice command through the remote control supporting the smart TV. At this time, the smart TV can directly make a voice response. It can be seen that when the user inputs a voice command in this way, the smart device 200 supporting the remote control can be directly determined as the response device, and generally there will be no problem of chaotic voice response.

[0057] When the user interacts through the far-field voice input method, multiple smart devices 200 with voice interaction functions can all detect the user's voice input and thus make responses respectively, resulting in chaotic voice responses. For this reason, in some embodiments, the user can also set different wake-up words for each smart device 200. For example, the wake-up word for the smart TV is "Hi! Little ×" and the wake-up word for the smart speaker is "Little × Little ×". When the user inputs a wake-up word, the smart device 200 corresponding to the wake-up word can be used as the response device to alleviate the problem of chaotic control.

[0058] However, this voice input method of determining the response device through different wake-up words not only requires the user to remember the correspondence between each device and the wake-up word, but also for smart devices 200 of the same manufacturer, since they have the same built-in smart voice system, their wake-up words are usually the same. Therefore, for such devices, there will still be a problem of chaotic control.

[0059] To alleviate the problem of control chaos, some embodiments of the present application provide an intelligent voice control method. The method can be applied to a smart home system composed of a server 400 and multiple smart devices 200 to avoid multiple smart devices 200 responding to the same voice command. To implement the intelligent voice control method, the server 400 in the smart home system should at least include a storage module 410, a communication module 420, and a control module 430. Among them, the storage module 410 is configured to store a mapping database, and the mapping database includes the mapping relationship between user intent information and device capability information. The communication module 420 is configured to establish a communication connection with multiple smart devices 200 to send control commands and related data to the multiple smart devices 200. The control module 430 is configured to execute the program steps on the server 400 side in the intelligent voice control method to determine the executing device among the multiple smart devices 200. Similarly, to meet the implementation of the intelligent voice control method, the smart device 200 in the smart home system should at least include an audio input device, an audio output device, a communicator 220, and a controller 250. Among them, the audio input device is configured to detect the voice audio data input by the user. The audio output device is configured to play a voice response. The communicator 220 is configured to establish a communication connection with the server 400 to upload control commands to the server 400 and receive the control commands sent by the server 400. The controller 250 is configured to execute the program steps on the smart device 200 side in the intelligent voice control method to complete the response of the intelligent voice control process.

[0060] As Figure 5 , Figure 6 shown, the intelligent voice control method includes the following:

[0061] The smart device 200 obtains the voice audio data input by the user. When the user is in the smart home system environment, voice input can be performed in real time. Then, the built-in audio input device of the smart device 200 can convert the voice sound signal input by the user into an electrical signal, and through a series of signal processing methods such as noise reduction, amplification, encoding, and conversion, obtain the voice audio data.

[0062] When performing voice interaction, the user can input voice audio data in various ways. That is, in some embodiments, the user can input voice audio data through the built-in audio input device of the smart device 200. For example, the user can input the voice "Hi! Xiao ×, I want to watch a movie" through the built-in microphone device on the smart device 200. Then, the microphone can convert the voice sound signal into an electrical signal and transmit it to the controller 250 for subsequent processing.

[0063] In order to trigger the smart device 200 to perform intelligent voice control, in some embodiments, the user can also include a specific wake-up word in the input voice audio data. The wake-up word is a voice containing specific content, such as "Hi! Xiao×", "Xiao× Xiao×", "Hey! ××", etc. For the process of the user inputting voice audio data, especially the process of inputting voice audio data through the built-in far-field microphone of the smart device 200, the smart device 200 can judge whether the voice input by the user contains the wake-up word. When the wake-up word is detected, subsequent processing is performed to alleviate the false triggering of the intelligent voice control process.

[0064] After receiving the user's voice and audio data, the smart device 200 can also generate voice control instructions based on the voice and audio data. Voice control instructions are control commands with a specific instruction format, including control action functions, control object codes, and other content. After receiving the voice and audio data, the smart device 200 can first convert the voice and audio data into text using the voice processing module in the intelligent voice system. Specifically, the waveform data in the voice and audio data is converted into text data through acoustic feature extraction.

[0065] After converting the unstructured text data into text data, the smart device 200 can use a word segmentation tool to convert the unstructured text data into structured text data. Specifically, the smart device 200 can remove meaningless text content such as modal particles and auxiliary words from the text data through methods such as word library matching, retain the keywords in the text data, and separate multiple keywords according to their meanings to obtain structured text.

[0066] After obtaining structured text data, the smart device 200 can also input the structured text into a text processing model. A text processing model is an artificial intelligence model based on machine learning. After inputting the text data, the text processing model can calculate the probability of the text information belonging to a specific semantic meaning. Therefore, by using various control instructions as classification labels, the text processing model can output the classification probability of each control instruction in the text data. The control instruction with the highest classification probability is the control instruction corresponding to the voice audio data.

[0067] The smart device 200 can repeatedly train the initial model using sample data and pre-set input and output rules to obtain a text processing model. The sample data is labeled text information. During model training, the sample data is used as input, and classification probabilities are output. The output is compared with the labels in the sample data to obtain a training error. This training error is then backpropagated, adjusting the model parameters based on the training error. By repeatedly inputting a large amount of sample data, a text processing model capable of accurately outputting recognition results can be obtained.

[0068] After model calculation, the intelligent device 200 can convert the voice audio data input by the user into a voice control instruction. After being converted by the intelligent device 200, the controlled device or the server 400 can directly process the voice control instruction after receiving it, such as executing a control action according to the voice control instruction and extracting the user's intention from the voice control instruction, etc.

[0069] Obviously, in some embodiments, the intelligent device 200 can directly send the voice audio data as a voice control instruction. That is, for an intelligent device 200 with relatively low data processing capabilities or without a built-in complete intelligent voice system, the intelligent device 200 can directly forward the audio data, and the server 400 or other intelligent devices 200 perform language processing to relieve the current computing load of the intelligent device 200.

[0070] After generating the voice control instruction, the intelligent device 200 can send the voice control instruction to the server 400 so that the server 400 can screen the intelligent devices 200 in the current smart home system according to the voice control instruction. In some embodiments, the intelligent device 200 can first detect the voice audio data input by the user to check whether it contains clear response device information. For example, when the voice instruction input by the user contains a wake-up word and only one intelligent device 200 supports using this wake-up word to perform voice interaction, it can be determined that the current voice audio data includes clear response device information. At this time, the intelligent device 200 does not have to report the voice control instruction to the server 400 for device screening, but directly passes the voice control instruction to the intelligent device 200 that uses this wake-up word through the local smart home system for voice control response.

[0071] Through the detection of the content of the voice audio data by the intelligent device 200, when it is determined that the voice audio data input by the user does not clearly indicate the response device, the intelligent device 200 then sends the voice control instruction to the server 400 to screen the response device through the server 400.

[0072] Since multiple intelligent devices 200 with built-in intelligent voice systems can be included in the smart home system, when a user inputs voice, multiple intelligent devices 200 in the smart home system can all detect the voice audio data. At this time, in order to avoid duplicate data transmission, after receiving a voice control instruction, the server 400 can pause the generation process and reporting process of voice control instructions in other intelligent devices 200. For example, after the smart TV sends a voice control instruction to the server 400, the server 400 can send control instructions for pausing instruction generation and instruction sending to the smart speaker and smart refrigerator in the smart home system where the smart TV is located. After receiving this control instruction, both the smart speaker and the smart refrigerator stop generating and sending voice control instructions.

[0073] In some embodiments, for accurate control, after receiving the voice control instruction sent by the intelligent device 200, the server 400 can also detect voice control instructions with the same content within a specific detection period. When multiple voice control instructions with the same content but from different intelligent devices 200 are received within the detection period, the server 400 can determine whether the multiple intelligent devices 200 belong to the same smart home system. If the multiple intelligent devices 200 belong to the same smart home system, the server 400 can perform a subsequent operation only once for the multiple voice control instructions with the same content. For multiple intelligent devices 200 belonging to different smart home systems, the server 400 needs to perform subsequent operations for each voice control instruction separately.

[0074] Obviously, in order to implement the judgment of voice control instructions, when generating a voice control instruction, the intelligent device 200 can add device identification information of the intelligent device 200 to the voice control instruction. The content that can be used as device identification information can include the device name, device model, device type, device network address, etc. of the intelligent device 200. The device identification information can be written into the voice control instruction through a specific coding rule. Specific information codes can also be written into the voice control instruction to replace the device identification information.

[0075] After receiving the voice control instruction sent by the intelligent device 200, the server 400 can extract user intent information from the voice control instruction. Among them, the user intent information can be determined by the control action and control object carried in the voice control instruction. For example, when the voice control instruction is "play a movie", it can be determined that the current user intent is to play a video.

[0076] Since multiple voice control instructions can be attributed to the same user intention, in order to reduce the amount of data matching, multiple intention topics for characterizing user intentions can be built into the server 400. After receiving a voice control instruction, the server 400 can determine the user intention topic to which the current voice control instruction belongs according to the control action and control object in the voice control instruction. For example, in the server 400, three user intention topics, namely video_topic, music_topic, and fridge_control_topic, can be stored. When the content of the received voice control instruction is "play a movie", it can be determined that the user intention to which the current voice control instruction belongs is video_topic. Similarly, when the content of the received voice control instruction is "play a TV drama", it can also be determined that the user intention to which the current voice control instruction belongs is video_topic. When the content of the received voice control instruction is "play music", it can be determined that the user intention to which the current voice control instruction belongs is music_topic.

[0077] The user intention information can also be obtained by processing through the voice processing model built into the server 400. That is, in some embodiments, after obtaining a voice control instruction, the server 400 can first convert the voice audio data into text information, and then input the text information into the text processing model to obtain the recognition result output by the text processing model for the text information, that is, obtain the classification probability of the text information for the preset user intention label. Then, the user intention label with the highest classification probability in the processing result is extracted as the user intention information. Among them, the text processing model is also an artificial intelligence model based on machine learning.

[0078] It can be seen that in this embodiment, the data processing capability of the server 400 can be used to process the voice audio data corresponding to the voice control instruction, so as to adapt to the situation where the voice control instruction is directly transmitted voice audio data of the intelligent device 200, and relieve the computing load of the intelligent device 200. In addition, a dynamic monitoring mechanism can be established to monitor the computing load status of the intelligent device 200 and the server 400 in real time, so as to select the intelligent device 200 or the server 400 with a good load status to perform text processing, so as to reasonably utilize computing resources. That is, if the computing load of the intelligent device 200 is light and the computing load of the server 400 is heavy, the intelligent device 200 performs text processing to generate a voice control instruction directly containing user intention information; if the computing load of the server 400 is light and the computing load of the intelligent device 200 is heavy, the intelligent device 200 directly sends the voice audio data to the server 400, and the server 400 performs text processing to determine the user intention information.

[0079] After extracting the user intention information, the server 400 can match the intelligent device 200 capable of implementing the user intention information in the mapping database to generate a target device list. Among them, the mapping database can record the mapping relationship between the user intention information and the device capability information, that is, for a specific user intention, which intelligent devices 200 have the device capabilities to implement the user intention.

[0080] The device capability information can also correspond to the user intention information and set multiple different capability themes. For example, for the user intention corresponding to the video_topic, the intelligent device 200 capable of implementing the user intention should support the device capability of video_play; for the user intention corresponding to the music_topic, the intelligent device 200 capable of implementing the user intention should support the device capability of music_play; for the user intention corresponding to the fridge_control_topic, the intelligent device 200 capable of implementing the user intention should support the device capability of fridge_control. And the same intelligent device 200 can support different capability themes according to its hardware conditions. For example, a smart TV can support both video_play and music_play.

[0081] After obtaining the user intention information, the server 400 can match the device capabilities that can meet the user intention in the mapping database, and then determine the intelligent device 200 capable of implementing the user intention through the support relationship between the device capabilities and the intelligent device 200.

[0082] In some embodiments, in order to generate a target device list, after extracting the user intention information, the server 400 can obtain the current network device table. The network device table includes all the intelligent device information in the smart home system where the intelligent device 200 is located. Then, by traversing the device capability information required to support the user intention information to obtain the target capability, and searching for the intelligent device 200 with the target capability in the network device table to generate a target device list.

[0083] For example, for the voice control instruction content of "I want to listen to music", it can be determined that the user intention is "music_topic", and the corresponding device capability that the intelligent device 200 needs to support is determined to be "music_play". And through the matching process in the current smart home system, it can be determined that the intelligent devices 200 with the "music_play" device capability include a smart TV and a smart speaker. Then the server 400 can generate a target device list including the smart TV and the smart speaker.

[0084] After obtaining the list of target devices, the server 400 can also perform a secondary screening on the target list according to the distance between the sound source position and the intelligent device 200, that is, the server 400 searches for an execution device in the target device list, where the execution device is the intelligent device 200 closest to the position where the voice control instruction is initiated.

[0085] In order to detect the distance between the position where the voice control instruction is initiated and the intelligent device 200, the intelligent devices 200 can report the user voice audio data collected by each of them to the server 400. A judgment module can be built into the server 400 to judge the sound source position or the distance between the sound source position and each intelligent device 200 according to the voice audio data reported by each intelligent device 200.

[0086] In some embodiments, the server 400 can obtain the voice control instructions reported by each intelligent device 200 in the target device list and extract the sound energy value from the voice control instructions. By comparing multiple sound energy values, the intelligent device 200 with the highest sound energy value is obtained, and thus the intelligent device 200 with the highest sound energy value is marked as the execution device.

[0087] The specific sound source position detection process can be calculated based on the time difference of arrival of the sound source and the ratio of direct sound energy to reverberant sound energy. Since the reverberation time parameter T60 of the same room is determined, that is, the time required for the energy to decay by 60 db at any position is the same, and T60 can be estimated based on the ratio of direct sound energy to reverberant sound energy at the corresponding position, therefore, based on the spectrogram formed by beamforming and the time difference of arrival of the sound source, the ratio of direct sound energy to reverberant sound energy of all intelligent devices 200 in the environment with respect to the sound source can be obtained, and then the direct energy can be obtained. The direct sound energies of the sound sources received by each device are arranged, and thus the intelligent device 200 closest to the sound source position can be judged.

[0088] In some embodiments, the detection process of the distance between the sound source position and the intelligent device 200 can also be completed by each intelligent device 200. That is, the intelligent device 200 can obtain an image of the current environment through a multi-camera, construct a three-dimensional space model based on the images at multiple angles, and then extract the human figure in the three-dimensional space model according to the image recognition method, so as to locate the position of the user in the three-dimensional space model, that is, the sound source position. After locating the sound source position, the intelligent device 200 then determines the distance between the sound source position and each intelligent device 200 according to the placement state of the current smart home model, and finally sends the calculated distance to the server 400, so that the server 400 can determine that the intelligent device 200 closest to the sound source position is the execution device.

[0089] In some embodiments, the detection of the distance between the sound source position and the intelligent device 200 can also be based on the time difference of the same voice audio data collected by multiple microphones to achieve the positioning of the sound source position. Since the propagation speed of the sound signal in the air is fixed and known, when multiple microphones detect the same voice audio data at different positions, the distance between the sound source position and the intelligent device 200 can be calculated based on the time difference of the detected starting moment of the sound and the positions of the intelligent devices 200. The above calculation process can be uniformly completed by the server 400, that is, multiple intelligent devices 200 can report the voice audio data detected by themselves to the server 400, and the server 400 extracts the moment when the voice is detected from each voice audio data, and then calculates the distance between the sound source position and the intelligent device 200.

[0090] Since the number and functions of the intelligent devices 200 included in different smart home systems are different, the number of intelligent devices 200 included in the target device list determined according to the matching result of the user intention and the device capabilities is also different. When the target device list includes different numbers of intelligent devices 200, the query process of the server 400 for the executing device can also be different. That is, as Figure 7 shown, in some embodiments, after generating the target device list, the server 400 can obtain the number of devices in the target device list.

[0091] If the number of devices is greater than 1, that is, after screening by the user intention and the device capabilities, multiple intelligent devices 200 can be screened out to meet the current user intention. Therefore, the server 400 can perform a secondary screening on the target device list based on the principle of proximity decision-making in the manner provided in the above embodiments, that is, perform the step of searching for the executing device in the target device list.

[0092] If the number of devices is equal to 0, that is, after matching the user intention and the device capabilities, it is determined that there is no intelligent device 200 in the current smart home system that meets the user intention, or the intelligent device 200 that meets the user intention is in a cable management state. At this time, the intelligent device 200 needs to make a prompt feedback for the voice audio data input by the user. Therefore, the server 400 can search for the executing device in the network device table, that is, query the intelligent device 200 closest to the sound source position in the current entire smart home system as the executing device to feedback the prompt voice through the executing device.

[0093] For example, when the user's input voice content is "lower the temperature inside the fridge", the user intention corresponding to this voice control instruction is fridge_control_topic. At this time, the server 400 can match the corresponding required device capabilities in the mapping database, that is, fridge_control. However, by detecting the online status of all intelligent devices 200 in the current smart home system, the intelligent device 200 with the fridge_control device capability is in the off state, and there is no other intelligent device 200 with this device capability. At this time, the server 400 cannot filter out the target device list, that is, the number of devices included in the target device list is 0. For this situation, the server 400 can search for the intelligent device 200 closest to the user's location in the entire smart home system as the execution device, that is, the display device, so as to play the feedback prompt voice "I don't understand what you're saying" or "The refrigerator is in the shutdown state, please turn it on" through the display device.

[0094] If the number of devices is equal to 1, that is, the target device list only contains one intelligent device 200 that can support the current user intention. Therefore, in order to reduce the data processing volume, the server 400 can skip the subsequent nearest decision method and directly mark the intelligent device 200 in the target device list as the execution device. For example, when the user inputs the voice "I want to watch a movie", the server 400 matches that the intelligent device 200 with the "video_play" capability according to the user intention is only the smart TV, that is, the target device list only includes one intelligent device 200, which is the smart TV. Therefore, the server 400 can directly use the smart TV as the execution device to play the voice "The following movies are found for you".

[0095] After querying and obtaining the execution device, the server 400 can send the initially obtained voice control instruction to the execution device so that the execution device can perform a voice response for the voice control instruction. For example, after the user inputs the voice "I want to listen to music", the server 400 can first filter out the target device list that supports this user intention according to the user intention "music_topic", that is, the "smart TV" and the "smart speaker". Then, according to the above distance detection process, detect the distances between the smart TV and the smart speaker and the sound source position, and determine the smart TV closest to the sound source position as the execution device, so as to send the voice control instruction to the smart TV, enabling the smart TV to respond to this voice control instruction with the feedback voice "Music has been played for you" and start the voice playback function to start playing music.

[0096] As can be seen from the above technical solutions, when there are multiple intelligent devices 200 that can respond to voice control instructions in the smart home system in the above embodiments, the target device list can be determined first by screening the user intention and device capabilities, and then according to the principle of proximity decision-making, the intelligent device 200 closest to the sound source position can be screened out from the target device list as the execution device. Finally, the voice control instruction is sent to the execution device so that the execution device can respond to the voice control instruction, and other intelligent devices 200 can remain silent, alleviating the mutual interference and control chaos problems caused by multiple devices responding simultaneously.

[0097] Since the intelligent device 200 needs to restart the relevant application when responding to some functions, and starting the relevant application takes a certain amount of time, it causes the user to wait for a long time to realize the relevant functions after inputting the voice control instruction. When the intelligent device 200 is in the appropriate playing state, the intelligent device 200 does not need to restart the relevant application and directly makes a voice control response, greatly reducing the waiting time of the user. Therefore, in order to improve the response speed of the voice interaction process, in some embodiments, when the server 400 searches for the execution device in the target device list according to the principle of proximity decision-making, it can detect the playing state of each intelligent device 200 to determine the intelligent device 200 in the appropriate playing state for preferentially responding to the user's voice interaction instruction.

[0098] That is, in the step of searching for the execution device in the target device list, the server 400 can obtain the playing state of each intelligent device in the target device list, and set the execution weight value for the intelligent device 200 according to the playing state, and add the execution weight value to the target device list.

[0099] The server 400 can set the execution weight value according to different user intentions or device capabilities. For example, when the device capability required by the user intention is "video_play", the execution weight value of the "ready" state is higher than that of the "power on" state. Therefore, the execution weight value of the intelligent device 200 in the "ready" state can be set to 1, and the execution weight value of the "power on" state is 0.5.

[0100] After adding the execution weight to the target device list as well, the server 400 can comprehensively determine the execution device based on the execution weight and the sound source localization result. That is, in the step of searching for the execution device in the target device list, the server 400 can first obtain the sound source localization information reported by each intelligent device in the target device list. Among them, the sound source localization information includes the distance between the intelligent device and the location where the voice control instruction is initiated. If the target device list includes the execution weight, the execution weight is corrected according to the sound source localization information, and then the intelligent device 200 with the largest execution weight is marked as the execution device.

[0101] For example, the server 400 can set a correction coefficient according to a preset distance range. In the distance range of 0 - 2m, the correction coefficient is 1. In the distance range of 2 - 4m, the correction coefficient is 0.8. In the distance range of 4 - 6m, the correction coefficient is 0.6... Therefore, in combination with the playback state, it can be determined that the execution weight corresponding to device A in the "ready" state and in the distance range of 2 - 4m is 0.8×1 = 0.8, while the execution weight corresponding to device B in the "power on" state and in the distance range of 0 - 2m is 1×0.5 = 0.5. That is, device A is determined as the execution device.

[0102] It can be seen that the above - mentioned embodiment's method of determining the execution device based on the principle of proximity decision - making and weighted information can perform weighted comparison on the basis of the original algorithm result, and select the intelligent device 200 with the maximum value as the optimal proximity wake - up as the execution device. Therefore, it can synthesize proximity response and response speed, improving the user's voice interaction experience.

[0103] In the above - mentioned embodiment, the server 400's first - screening process for the intelligent devices 200 in the current smart home system is based on the matching result of the user's intention and the device capabilities in the mapping database. Obviously, in order to improve the accuracy of the first - screening process, when a new device is online in the smart home system, the server 400 needs to update the mapping database according to the device capabilities of the online device.

[0104] That is, as Figure 8 shown, in some embodiments, the server 400 can receive the online request sent by the intelligent device, and in response to the online request, obtain the identification information of the intelligent device. Then, according to the identification information, match the device capability information supported by the intelligent device 200, so as to store the identification information of the intelligent device 200 and the device capability information supported by the intelligent device 200, and update the mapping database.

[0105] For example, when a new smart refrigerator is connected to the current smart home system, the smart refrigerator needs to send an online request to the server 400. The online request may include the identification information of the smart refrigerator, that is, the device model is "H××-BCD-×××WTDVBPV". The server 400 then looks up the device capabilities supported by the smart refrigerator according to the device model, that is, "video_play", "music_play", and "fridge_control", and stores the found device capabilities, device model, and other contents in the mapping database, thereby updating the mapping database.

[0106] Based on the above intelligent voice control method, in some embodiments of the present application, a server 400 is further provided. As Figure 9 shown, the server 400 includes: a storage module 410, a communication module 420, and a control module 430. Among them, the storage module 410 is configured to store a mapping database, and the mapping database includes the mapping relationship between user intent information and device capability information; the communication module 420 is configured to establish a communication connection with multiple smart devices; the control module 430 is configured to execute the following program steps:

[0107] Receive a voice control instruction sent by a smart device;

[0108] In response to the voice control instruction, extract user intent information from the voice control instruction;

[0109] Generate a target device list, and the target device list includes smart devices that can implement user intent information obtained by matching in the mapping database;

[0110] Find an execution device in the target device list, and the execution device is the smart device closest to the location where the voice control instruction is initiated;

[0111] Send the voice control instruction to the execution device to trigger the execution device to perform a voice response for the voice control instruction.

[0112] In cooperation with the above server 400, in some embodiments, a smart device 200 is further provided. As Figure 10 shown, the smart device 200 includes: an audio input device, an audio output device, a communicator 220, and a controller 250. Among them, the audio input device is configured to detect voice audio data input by a user; the audio output device is configured to play a voice response; the communicator 220 is configured to establish a communication connection with the server 400; the controller 250 is configured to execute the following program steps:

[0113] Obtain voice audio data input by a user;

[0114] Generate a voice control instruction according to the voice audio data;

[0115] Send a voice control instruction to the server, so that the server extracts user intention information from the voice control instruction, and matches intelligent devices that can implement the user intention information in the mapping database to generate a target device list; and search for an execution device in the target device list to issue the voice control instruction;

[0116] Perform a voice response for the voice control instruction.

[0117] As can be seen from the above technical solutions, the server 400 and the intelligent device 200 provided in the above embodiments can form a smart home system. The smart home system can, after receiving a voice control instruction input by the user, first send the voice control instruction to the server 400. The server 400 extracts user intention information from the voice control instruction, and matches intelligent devices that can implement the user intention information in the mapping database to generate a target device list. Then, search for the execution device closest to the user or with the highest execution weight in the target device list, and issue the voice control instruction to the execution device to respond to the voice control instruction input by the user.

[0118] The method can determine the execution device for responding to the user's voice control instruction based on the user intention matching result and the sound source localization result. Thus, when there are multiple intelligent devices 200 in the smart home system that can respond to voice control instructions, only the execution device makes a response, alleviating the problems of control chaos and the situation where only the execution device responds but the feedback cannot support the control service.

[0119] For the similar parts between the embodiments provided in this application, reference can be made to each other. The specific embodiments provided above are only several examples under the general concept of this application, and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation manner extended based on the solution of this application without creative efforts belongs to the protection scope of this application.

Claims

1. A server, characterized in that, Including: A storage module configured to store a mapping database, which includes the mapping relationship between user intent information and device capability information; A communication module configured to establish communication connections with multiple intelligent devices; A control module configured to: Receive a voice control instruction sent by an intelligent device, where the voice control instruction is generated by the intelligent device based on voice audio data input by a user, and the voice audio data does not contain response device information; In response to the voice control instruction, extract user intent information from the voice control instruction; Generate a target device list, which includes intelligent devices that can implement the user intent information obtained by matching in the mapping database; Obtain the playback status of each intelligent device in the target device list; Set an execution weight for the intelligent device according to the playback status; Add the execution weight to the target device list; Obtain the sound source localization information reported by each intelligent device in the target device list, where the sound source localization information includes the distance between the intelligent device and the location where the voice control instruction is initiated; Correct the execution weight according to the sound source localization information; Mark the intelligent device with the largest execution weight as the execution device; Send the voice control instruction to the execution device to trigger the execution device to perform a voice response for the voice control instruction.

2. The server according to claim 1, wherein The voice control instruction includes voice audio data input by a user, and the control module is further configured to: In the step of extracting user intent information from the voice control instruction, convert the voice audio data into text information; Input the text information into a text processing model, where the text processing model is an artificial intelligence model based on machine learning; Obtain the recognition result output by the text processing model for the text information, where the recognition result is the classification probability of the text information for a preset user intent label; Extract the user intent label with the highest classification probability in the recognition result as the user intent information.

3. The server according to claim 1, characterized in that The control module is further configured to: Receive an online request sent by the intelligent device; In response to the online request, obtain the identification information of the intelligent device; Match the device capability information supported by the intelligent device according to the identification information; Store the identification information of the intelligent device and the device capability information supported by the intelligent device.

4. The server according to claim 1, characterized in that, The control module is further configured to: In the step of generating the target device list, obtain the current network device table, where the network device table includes all intelligent devices in the smart home system where the intelligent device is located; Traverse the device capability information required to support the user intent information to obtain the target capability; Search for intelligent devices with the target capability in the network device table to generate the target device list.

5. The server according to claim 4, wherein The control module is further configured to: After the step of generating the target device list, obtain the number of devices in the target device list; If the number of devices is equal to 0, search for the execution device in the network device table; If the number of the devices is equal to 1, mark the smart device in the target device list as the execution device; If the number of the devices is greater than 1, execute the step of obtaining the playing states of the smart devices in the target device list.

6. The server according to claim 1, characterized in that The control module is further configured as: In the step of marking the smart device as the execution device, obtain the voice control instructions reported by each smart device in the target device list; Extract the sound energy value from the voice control instructions; Compare the sound energy values to obtain the smart device with the highest sound energy value; Mark the smart device with the highest sound energy value as the execution device.

7. An intelligent device, characterized in that, Comprising: An audio input device, configured to detect the voice audio data input by the user; An audio output device, configured to play the voice response; A communicator, configured to establish a communication connection with the server; A controller, configured as: Obtain the voice audio data input by the user; Generate a voice control instruction according to the voice audio data; If the response device information does not exist in the voice audio data, send the voice control instruction to the server, so that the server extracts the user intention information from the voice control instruction, matches the smart devices capable of implementing the user intention information in the mapping database to generate a target device list; and find the execution device in the target device list to send the voice control instruction, wherein the voice control instruction is further used to enable the server to obtain the playing states of the smart devices in the target device list; set the execution weight value for the smart devices according to the playing states; add the execution weight value to the target device list; obtain the sound source localization information reported by the smart devices in the target device list, where the sound source localization information includes the distance between the smart device and the voice control instruction initiation position; correct the execution weight value according to the sound source localization information; mark the smart device with the largest execution weight value as the execution device; Execute a voice response for the voice control instruction.

8. An intelligent voice control method, characterized in that, Applied to a smart home system, the smart home system includes a server and smart devices, and a communication connection is established between the server and the smart devices; the intelligent voice control method includes: The smart device obtains the voice audio data input by the user and generates a voice control instruction according to the voice audio data; If the response device information does not exist in the voice audio data, the smart device sends the voice control instruction to the server; The server extracts the user intention information from the voice control instruction and matches the smart devices capable of implementing the user intention information in the mapping database to generate a target device list; The server obtains the playback status of each intelligent device in the target device list; sets an execution weight for the intelligent device according to the playback status; adds the execution weight to the target device list; obtains the sound source localization information reported by each intelligent device in the target device list, where the sound source localization information includes the distance between the intelligent device and the voice control instruction initiation location; corrects the execution weight according to the sound source localization information; marks the intelligent device with the largest execution weight as the execution device, and sends a voice control instruction to the execution device; The intelligent device serving as the execution device performs a voice response to the voice control instruction.

Images