Intelligent Home Appliance Terminal Control Method, Device and Storage Medium Combining Auditory and Visual

Abstract

The present application discloses an intelligent home appliance terminal control method, device and storage medium combining vision and audition, which relates to the technical field of smart home / intelligent home. The method includes: obtaining a control instruction, obtaining control information corresponding to the control instruction, and sending the control information to a server so that the server performs intention recognition on the control information to obtain a recognition result, and determining first information to be used according to the recognition result, receiving the first information to be used sent by the server, and processing the first information to be displayed and the first information to be played when the first information to be used includes the first information to be displayed and the first information to be played, wherein there is a first corresponding relationship between the first information to be displayed and the first information to be played. In this way, the user can intuitively obtain the information corresponding to the control instruction from both the visual and auditory aspects, which can improve the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of smart home / smart family, and particularly to a control method, device and storage medium for an intelligent home appliance terminal with audio-visual combination. Background Art

[0002] At present, intelligent home appliance terminals account for an increasingly high proportion in people's daily lives. Users can control intelligent home appliance terminals, so that the intelligent home appliance terminals can respond to users' instructions to meet users' usage requirements.

[0003] However, current intelligent home appliance terminals cannot clearly display the control results of responding to users' instructions to users, resulting in users being unable to intuitively obtain the control results and reducing users' usage experience. Summary of the Invention

[0004] The present application provides a control method, device and storage medium for an intelligent home appliance terminal with audio-visual combination, which are used to solve the defect that in the prior art, the control results of responding to users' instructions cannot be clearly displayed to users, resulting in users being unable to intuitively obtain the control results and reducing users' usage experience, and to improve users' usage experience.

[0005] The present application provides a control method for an intelligent home appliance terminal with audio-visual combination. The method is applied to an intelligent home appliance terminal and includes:

[0006] Obtain a control instruction;

[0007] Obtain control information corresponding to the control instruction, and send the control information to a server, so that the server performs intention recognition on the control information to obtain a recognition result, and determines first information to be used according to the recognition result;

[0008] Receive the first information to be used sent by the server;

[0009] When the first information to be used includes first information to be displayed and first information to be played, process the first information to be displayed and the first information to be played, where there is a first corresponding relationship between the first information to be displayed and the first information to be played.

[0010] According to the control method for an intelligent home appliance terminal with audio-visual combination provided by the present application, after the step of receiving the first information to be used sent by the server, the method further includes:

[0011] When the first information to be used includes control parameters, execute the control parameters and generate a return value according to the execution situation;

[0012] Send the return value to the server so that the server determines the second information to be used based on the return value;

[0013] Receive the second information to be used sent by the server;

[0014] When the second information to be used includes second information to be displayed and second information to be played, process the second information to be displayed and the second information to be played, where there is a second corresponding relationship between the second information to be displayed and the second information to be played.

[0015] According to an intelligent home appliance terminal control method combining vision and audition provided by the present application, the step of processing the first information to be displayed and the first information to be played includes:

[0016] Determine a corresponding first display method according to the type of the first corresponding relationship;

[0017] When performing playback processing on the first information to be played, perform display processing on the first information to be displayed by using the first display method;

[0018] The step of processing the second information to be displayed and the second information to be played includes:

[0019] Determine a corresponding second display method according to the type of the second corresponding relationship;

[0020] When performing playback processing on the second information to be played, perform display processing on the second information to be displayed by using the second display method.

[0021] According to an intelligent home appliance terminal control method combining vision and audition provided by the present application, the first information to be displayed includes first address information, and the second information to be displayed includes second address information;

[0022] The step of performing display processing on the first information to be displayed by using the first display method includes:

[0023] Obtain a corresponding first video from the pre-stored videos according to the first address information;

[0024] Display the first video by using the first display method;

[0025] The step of performing display processing on the second information to be displayed by using the second display method includes:

[0026] Obtain a corresponding second video from the pre-stored videos according to the second address information;

[0027] Display the second video using the second display method.

[0028] According to an intelligent home appliance terminal control method combining vision and sound provided by the present application, after the step of obtaining the control information corresponding to the control instruction, the method further includes:

[0029] Determine the corresponding usage information according to the control information corresponding to the control instruction and the pre-stored correspondence between the control information and the usage information, and use the corresponding usage information as the third information to be used, where the third information to be used includes third information to be displayed and third information to be played, and there is a correspondence between the third information to be displayed and the third information to be played;

[0030] Process the third information to be displayed and the third information to be played.

[0031] The present application also provides an intelligent home appliance terminal control method combining vision and sound. The method is applied to a server and includes:

[0032] Receive the control information sent by the intelligent home appliance terminal, where the control information is the information corresponding to the control instruction obtained by the intelligent home appliance terminal after obtaining the control instruction;

[0033] Perform intention recognition on the control information to obtain a recognition result, and determine the first information to be used according to the recognition result;

[0034] Send the first information to be used to the intelligent home appliance terminal, so that when the first information to be used includes first information to be displayed and first information to be played, the intelligent home appliance terminal processes the first information to be displayed and the first information to be played, where there is a first correspondence between the first information to be displayed and the first information to be played.

[0035] According to an intelligent home appliance terminal control method combining vision and sound provided by the present application, after the step of sending the first information to be used to the intelligent home appliance terminal, the method further includes:

[0036] Receive the return value sent by the intelligent home appliance terminal, where the return value is a value generated by the intelligent home appliance terminal according to the execution situation after executing the control parameter when the first information to be used includes the control parameter.

[0037] Determine the second information to be used based on the return value, and send the second information to be used to the smart home appliance terminal, so that when the second information to be used includes second information to be displayed and second information to be played, the smart home appliance terminal processes the second information to be displayed and the second information to be played, where there is a second corresponding relationship between the second information to be displayed and the second information to be played.

[0038] This application also provides a control device for a smart home appliance terminal with audio-visual combination. The device is applied to the smart home appliance terminal and includes:

[0039] A first acquisition module, configured to acquire a control instruction;

[0040] A second acquisition module, configured to acquire control information corresponding to the control instruction, and send the control information to the server, so that the server performs intent recognition on the control information to obtain a recognition result, and determines first information to be used according to the recognition result;

[0041] A first receiving module, configured to receive the first information to be used sent by the server;

[0042] A first processing module, configured to process the first information to be displayed and the first information to be played when the first information to be used includes the first information to be displayed and the first information to be played, where there is a first corresponding relationship between the first information to be displayed and the first information to be played.

[0043] This application also provides a control device for a smart home appliance terminal with audio-visual combination. The device is applied to the server and includes:

[0044] A second receiving module, configured to receive control information sent by the smart home appliance terminal, where the control information is the information corresponding to the control instruction acquired by the smart home appliance terminal after acquiring the control instruction;

[0045] An identification module, configured to perform intent recognition on the control information to obtain a recognition result, and determine first information to be used according to the recognition result;

[0046] A sending module, configured to send the first information to be used to the smart home appliance terminal, so that the smart home appliance terminal processes the first information to be displayed and the first information to be played when the first information to be used includes the first information to be displayed and the first information to be played, where there is a first corresponding relationship between the first information to be displayed and the first information to be played.

[0047] The present application also provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to execute the intelligent home appliance terminal control method combining vision and audition as described in any one of the above through the computer program.

[0048] The present application also provides a computer-readable storage medium. The computer-readable storage medium includes a stored program, wherein the program, when running, executes to implement the intelligent home appliance terminal control method combining vision and audition as described in any one of the above.

[0049] The intelligent home appliance terminal control method, device and storage medium provided by the present application obtain a control instruction, obtain control information corresponding to the control instruction, and send the control information to a server, so that the server performs intention recognition on the control information to obtain a recognition result, and determines first information to be used according to the recognition result, receives the first information to be used sent by the server, and processes the first information to be displayed and the first information to be played when the first information to be used includes the first information to be displayed and the first information to be played, wherein there is a first corresponding relationship between the first information to be displayed and the first information to be played.

[0050] In this way, the intelligent home appliance terminal can process the first information to be displayed and the first information to be played. For the user, after sending the control instruction, the user can see the first information to be displayed on the intelligent home appliance terminal and hear the first information to be played, that is, the user can see the visual information corresponding to the control instruction displayed on the intelligent home appliance terminal and can hear the auditory information corresponding to the control instruction played by the intelligent home appliance. The user can intuitively obtain the information corresponding to the control instruction from both the visual and auditory aspects, which can improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present application and used together with the description to explain the principles of the present application.

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0053] Figure 1 is a schematic diagram of the hardware environment of an intelligent home appliance terminal control method combining vision and audition according to an embodiment of the present application;

[0054] Figure 2 is one of the flow diagrams of the intelligent home appliance terminal control method combining vision and audition provided by the present application;

[0055] Figure 3 It is the second flow schematic diagram of the intelligent home appliance terminal control method combining vision and audition provided by this application;

[0056] Figure 4 It is the third flow schematic diagram of the intelligent home appliance terminal control method combining vision and audition provided by this application;

[0057] Figure 5 It is one of the structural schematic diagrams of the intelligent home appliance terminal control device combining vision and audition provided by this application;

[0058] Figure 6 It is the second structural schematic diagram of the intelligent home appliance terminal control device combining vision and audition provided by this application;

[0059] Figure 7 It is the structural schematic diagram of the electronic device provided by this application. Detailed implementation manners

[0060] In order to enable those skilled in the art to better understand the solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0061] It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0062] According to one aspect of the embodiments of this application, an intelligent home appliance terminal control method combining vision and audition is provided. This intelligent home appliance terminal control method combining vision and audition is widely applied to whole-house intelligent digital control application scenarios such as Smart Home, smart home, intelligent home appliance ecosystem, and Intelligence House ecosystem. Optionally, in this embodiment, the above-mentioned intelligent home appliance terminal control method combining vision and audition can be applied to, for exampleFigure 1 In the hardware environment composed of the terminal device 102 and the server 104 as shown. As Figure 1 shown, the server 104 is connected to the terminal device 102 through a network, and can be used to provide services (such as application services, etc.) for the terminal or the client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data operation services for the server 104.

[0063] The above network can include but is not limited to at least one of the following: wired network, wireless network. The above wired network can include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network. The above wireless network can include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 can be a smart home appliance terminal or an application terminal. When the terminal device 102 is an application terminal, the terminal device 102 is not limited to a PC, mobile phone, tablet computer, etc. When the terminal device 102 is a smart home appliance terminal, the terminal device 102 is not limited to a smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projection device, smart TV, smart clothes hanger, smart curtain, smart audio and video, smart socket, smart speaker, smart sound box, smart fresh air device, smart kitchen and bathroom equipment, smart bathroom equipment, smart floor sweeping robot, smart window cleaning robot, smart mopping robot, smart air purification device, smart steam box, smart microwave oven, smart kitchen water heater, smart purifier, smart water dispenser, smart door lock, etc.

[0064] In order to be able to intuitively display the control result corresponding to the instruction to the user, thereby improving the user experience, the embodiments of the present application provide an intelligent home appliance terminal control method, device, computer-readable storage medium, electronic device and computer program product that combine vision and sound. The following combines Figure 2 The first intelligent home appliance terminal control method provided by the embodiments of the present application.

[0065] As Figure 2 shown, the embodiments of the present application provide the first intelligent home appliance terminal control method that combines vision and sound. The method is applied to an intelligent home appliance terminal, and the method includes:

[0066] S201, obtain a control instruction.

[0067] When the user needs to set the smart home appliance terminal, a control instruction can be sent to the smart home appliance terminal. In this way, the smart home appliance terminal can obtain the control instruction that the user wants to use to control the smart home appliance terminal.

[0068] In one implementation, the smart home appliance terminal can collect the voice control instruction issued by the user through an audio collection device. That is to say, the control instruction can be a voice control instruction. In another implementation, the smart home appliance terminal can collect the gesture control instruction through an image collection device. That is to say, the control instruction can be a gesture control instruction. In yet another implementation, the smart home appliance terminal can receive the control instruction sent by the application terminal. That is to say, the control instruction can be the control instruction issued by the user through the application terminal.

[0069] S202, obtain the control information corresponding to the control instruction, and send the control information to the server, so that the server performs intent recognition on the control information to obtain an identification result, and determines the first information to be used according to the identification result.

[0070] After obtaining the control instruction, the control information corresponding to the control instruction can be obtained. There are various ways to obtain the control information. The following introduces the ways to obtain the control information.

[0071] The first acquisition method: when the control instruction is a voice control instruction, the smart home appliance terminal can use the audio corresponding to the voice control instruction as the control information, and thus send the control information to the server. After receiving the audio (control information) corresponding to the voice control instruction, the server can convert the control information into text information, perform intent recognition on the text information to obtain an identification result, and then determine the first information to be used according to the identification result. And send the first information to be used to the smart home appliance terminal.

[0072] The second acquisition method: when the control instruction is a voice control instruction, the smart home appliance terminal can convert the voice control instruction into text information, and use the text information as the control information corresponding to the control instruction. Send the text information (control information) to the server, and the server can perform intent recognition on the received text information (control information) to obtain an identification result. Then, the first information to be used can be determined according to the identification result. And send the first information to be used to the smart home appliance terminal.

[0073] The user will issue a voice control instruction within the preset range of the smart home appliance terminal. By adopting the second acquisition method, that is, the method of converting the voice control instruction into text information through the smart home appliance terminal, more accurate text information can be obtained to improve the accuracy of subsequent intent recognition.

[0074] For the third acquisition method, when the control instruction is a gesture control instruction, the smart home appliance terminal can use the video corresponding to the gesture control instruction as control information, and thus send the control information to the server. After receiving the video (control information) corresponding to the gesture control instruction, the server can perform intention recognition on the video corresponding to the gesture control instruction to obtain a recognition result, and then can determine the first information to be used according to the recognition result. And send the first information to be used to the smart home appliance terminal.

[0075] For the fourth acquisition method, when the control instruction is a gesture control instruction, the smart home appliance terminal can obtain multiple gesture key frames from the video corresponding to the gesture control instruction as control information. Thus, the gesture key frames (control information) are sent to the server. After receiving the gesture key frames (control information), the server can sort the gesture key frames according to the acquisition time corresponding to each gesture key frame, and then perform intention recognition on the sorted gesture key frames to obtain a recognition result, and then can determine the first information to be used according to the recognition result. And send the first information to be used to the smart home appliance terminal.

[0076] Compared with the third acquisition method, the fourth acquisition method can save network resources, so that the server can obtain control information more quickly, so as to be able to implement the control of the smart home appliance terminal in an audiovisual combination manner more quickly subsequently.

[0077] For the fifth acquisition method, when the control instruction is a control instruction sent by the application terminal, the smart home appliance terminal can use the parameter information corresponding to the control instruction as control information, and thus send the parameter information (control information) to the server. After receiving the parameter information (control information), the server can perform intention recognition on the parameter information to obtain a recognition result, and then can determine the first information to be used according to the recognition result. And send the first information to be used to the smart home appliance terminal.

[0078] As an implementation manner, the server can determine the first information to be used according to the recognition result. When the recognition result indicates that the intention corresponding to the control instruction can be recognized, the corresponding first information to be used can be determined according to the recognition result, and then the first information to be used is sent to the smart home appliance terminal.

[0079] When the recognition result indicates that the intention corresponding to the control instruction cannot be recognized, the words and display information indicating that the control instruction cannot be executed corresponding to the recognition result can be determined, and the words and display information are used as the first information to be used, and then the first information to be used is sent to the smart home appliance terminal.

[0080] S203, receive the first information to be used sent by the server.

[0081] S204. When the first information to be used includes the first information to be displayed and the first information to be played, process the first information to be displayed and the first information to be played.

[0082] The intelligent home appliance terminal can receive the first information to be used sent by the server. Among them, the first information to be used can include the first information to be displayed and the first information to be played. Among them, there is a first corresponding relationship between the first information to be displayed and the first information to be played.

[0083] When the first information to be used includes the first information to be displayed and the first information to be played, the first information to be displayed and the first information to be played can be processed. Specifically, the first information to be played can be played first according to the actual usage requirements, and then the first information to be displayed can be displayed.

[0084] The first information to be displayed can carry playback identification information. During the process of playing the first information to be played by the intelligent home appliance terminal, if it plays to the position corresponding to the playback identification information, the first information to be displayed can be correspondingly displayed.

[0085] Similarly, the first information to be displayed can be displayed first, and then the first information to be played can be played. The first information to be played can carry display identification information. During the process of displaying the first information to be displayed by the intelligent home appliance terminal, if it displays to the position corresponding to the display identification information, the first information to be played can be correspondingly played.

[0086] It is also possible to synchronously display the first information to be played and the first information to be displayed, and it is also possible to process the first information to be displayed and the first information to be played according to the playback and display order preset by the user. These are all reasonable.

[0087] When the recognition result indicates that the intention corresponding to the control instruction can be recognized, the first information to be displayed can include the information to be displayed corresponding to the type of the control instruction (the first information to be displayed) and the information to be played corresponding to the type of the control instruction (the first information to be played). The type of the control instruction can be a query instruction, an adjustment instruction, a call instruction, etc. Specifically, it can be set according to the actual situation.

[0088] For example, when the control instruction is to query the weather, the first information to be displayed can be the weather card-type information corresponding to the period queried by the user, and the first information to be played can be the voice information of the weather corresponding to the period queried by the user. The control instruction can also be instructions such as querying recipes and querying songs.

[0089] For another example, when the control instruction is an intelligent question and answer, the first information to be displayed can be the text corresponding to the answer, and the first information to be played can be the voice corresponding to the answer. The control instruction can be an instruction to query network information through the Internet, or an instruction to query the status of the intelligent home appliance terminal.

[0090] When the recognition result indicates that the intention corresponding to the control instruction cannot be recognized, the first information to be displayed can be the display information indicating that the control instruction cannot be executed, and the first information to be played can be the text information indicating that the control instruction cannot be executed. For example, the first information to be displayed can be the video corresponding to the inexecutability of the control instruction, and the first information to be played can be "The execution of the control instruction fails".

[0091] It can be seen that in this embodiment, the intelligent home appliance terminal can process the first information to be displayed and the first information to be played. For the user, after sending the control instruction, the user can see the first information to be displayed on the intelligent home appliance terminal and hear the first information to be played, that is, the user can see the visual information corresponding to the control instruction displayed on the intelligent home appliance terminal and can hear the auditory information corresponding to the control instruction played by the intelligent home appliance. The user can intuitively obtain the information corresponding to the control instruction from both the visual and auditory aspects, which can improve the user experience.

[0092] As an implementation manner of the embodiment of the present application, after the step of receiving the first information to be used sent by the server, the above steps may further include:

[0093] When the first information to be used includes control parameters, execute the control parameters and generate a return value according to the execution situation. Send the return value to the server so that the server determines the second information to be used based on the return value.

[0094] When the control instruction is an adjustment instruction, that is, when the control instruction is an instruction to adjust the status of the intelligent home appliance terminal, the first information to be used received by the intelligent home appliance terminal may include control parameters.

[0095] When the first information to be used includes control parameters, the intelligent home appliance terminal can execute the control parameters and generate a return value according to the execution of the control parameters. That is to say, the return value is a value that can characterize the execution situation.

[0096] Receive the second information to be used sent by the server. When the second information to be used includes the second information to be displayed and the second information to be played, process the second information to be displayed and the second information to be played.

[0097] The intelligent home appliance terminal can send the return value to the server. When the server receives the return value, it can determine the second information to be used according to the return value. Among them, the second information to be used can include the second information to be displayed and the second information to be played, and there is a corresponding relationship between the second information to be displayed and the second information to be played.

[0098] When the second information to be used includes the second information to be displayed and the second information to be played, the second information to be displayed and the second information to be played can be processed. For the specific processing method, reference can be made to the above description of the processing of the first information to be displayed and the first information to be played, which will not be elaborated here.

[0099] In this way, the intelligent home appliance terminal can process the second information to be displayed and the second information to be played. For the user, after sending a control instruction, the user can see the second information to be displayed on the intelligent home appliance terminal and hear the second information to be played, that is, the user can see the visual information corresponding to the control instruction displayed on the intelligent home appliance terminal and can hear the auditory information corresponding to the control instruction played by the intelligent home appliance. The user can intuitively obtain the information corresponding to the control instruction from both the visual and auditory aspects, which can improve the user experience.

[0100] To more conveniently understand the intelligent home appliance terminal control method combining vision and hearing provided in the embodiments of the present application, the following describes various situations corresponding to the first information to be used and the second information to be used.

[0101] In the first case, when the first information to be used does not include control parameters, the second information to be used does not exist. In this case, the first information to be used can include the first information to be displayed and the first information to be played.

[0102] Among them, the first information to be displayed can be a card-style picture, can also be a video, or can also be text that needs to be displayed. The first information to be played can be audio, or can also be text that needs to be played.

[0103] In the second case, the first information to be used only includes control parameters, and the second information to be used can include the second information to be displayed and the second information to be played. Among them, the second information to be displayed can be a card-style picture, can also be a video, or can also be text that needs to be displayed. The second information to be played can be audio, or can also be text that needs to be played.

[0104] When the second information to be played is text that needs to be played, corresponding voice can be generated based on the text, so as to play the voice corresponding to the second information to be played. Compared with the case where the second information to be played is audio, the case where the second information to be played is text can reduce the use of network resources, and the server can send the second information to be played to the intelligent home appliance terminal more quickly.

[0105] In the third case, the first information to be used includes control parameters, and the first information to be played and / or the first information to be displayed. In this case, the first information to be played can be information such as "adjusting" or "setting X mode". The first information to be displayed can be a picture or video representing the adjustment in progress. These are all reasonable. The second information to be used can include the second information to be displayed and the second information to be played.

[0106] In the embodiments of the present application, the first information to be used may include the first information to be displayed and / or the first information to be played. The second information to be used may also include the second information to be displayed and / or the second information to be played.

[0107] As an implementation manner of the embodiments of the present application, the steps of processing the first information to be displayed and the first information to be played may include:

[0108] Determine the corresponding first display manner according to the type of the first corresponding relationship. When performing playback processing on the first information to be played, perform display processing on the first information to be displayed by using the first display manner.

[0109] The type of the first corresponding relationship may be supplementary, explanatory, and simplified. Among them, the meaning of supplementary is that the first information to be played and the first information to be displayed can complement each other, so as to facilitate the user to clearly obtain the corresponding information.

[0110] The meaning of explanatory is that the first information to be played and the first information to be displayed can explain each other. In this type, the first information to be displayed can display the content that cannot be clearly explained by the first information to be played in the form of pictures or videos, etc., to facilitate the user's understanding.

[0111] The meaning of simplified is that the first information to be displayed is a simplification of the first information to be played, and the first information to be displayed may be the key information in the first information to be played, so as to facilitate the user to more intuitively obtain the key information.

[0112] When the type of the first corresponding relationship is supplementary, the corresponding first display mode can be a supplementary display mode. Specifically, when the first display mode is a supplementary display mode, in one implementation, when playing the first information to be played, the first information to be displayed can be displayed after the first information to be played is finished playing.

[0113] In another implementation, there can be multiple first information to be played, and correspondingly there are multiple first information to be displayed. The first information to be played can be played in sequence. For each first information to be played, after the first information to be played is finished playing, the first information to be displayed corresponding to the first information to be played can be displayed. After the corresponding first information to be displayed is finished displaying, the next first information to be played can be played in sequence, and after the next first information to be played is finished playing, the first information to be displayed corresponding to the next first information to be played can be displayed until the first information to be displayed at the last position in the sequence is finished displaying.

[0114] When the type of the first corresponding relationship is explanatory, the corresponding first display mode can be an explanatory display mode. When the first display mode is an explanatory display mode, in one implementation, when playing the first information to be played, the first information to be displayed can be synchronously displayed.

[0115] In another implementation, there can be multiple first information to be played, and correspondingly there are multiple first information to be displayed. The first information to be played can be played in sequence. For each first information to be played, when the first information to be played is being played, the first information to be displayed corresponding to the first information to be played can be synchronously displayed. After the corresponding first information to be displayed is finished displaying and the first information to be played is finished playing, the next first information to be played can be played in sequence, and when the next first information to be played is being played, the first information to be displayed corresponding to the next first information to be played can be synchronously displayed until the first information to be displayed at the last position in the sequence is finished displaying, and the first information to be played at the last position in the sequence is finished playing.

[0116] Among them, for each first information to be played, the duration of playing the first information to be played and the duration of displaying the corresponding first information to be displayed can be the same or different. It can be specifically set according to the actual usage situation.

[0117] When the type of the first corresponding relationship is the simplified type, the corresponding first display mode can be the simplified display mode. In this case, the first information to be displayed is text. In one implementation, when playing the first information to be played, all the text included in the first information to be displayed can be displayed, and when the first information to be played reaches the text belonging to the first information to be displayed, this text can be highlighted. Among them, the highlighting method can be enlarged display or highlighted display.

[0118] In another implementation, when playing the first information to be played, for each sentence included in the information to be played, when playing to this sentence, the key information (the text included in the first information to be displayed) included in this sentence can be correspondingly displayed.

[0119] The steps of processing the second information to be displayed and the second information to be played can include:

[0120] According to the type of the second corresponding relationship, determine the corresponding second display mode; when performing the playback processing on the second information to be played, use the second display mode to perform the display processing on the second information to be displayed.

[0121] The type of the second corresponding relationship can be the supplementary type, the explanatory type, and the simplified type. Among them, the meaning of the supplementary type is that the second information to be played and the second information to be displayed can complement each other, so as to facilitate the user to clearly obtain the corresponding information.

[0122] The meaning of the explanatory type is that the second information to be played and the second information to be displayed can explain each other. In this type, the second information to be displayed can display the content that the second information to be played cannot clearly explain in the form of pictures or videos, etc., to facilitate the user's understanding.

[0123] The meaning of the simplified type is that the second information to be displayed is a simplification of the second information to be played, and the second information to be displayed can be the key information in the second information to be played, so as to facilitate the user to more intuitively obtain the key information.

[0124] When the type of the second corresponding relationship is the supplementary type, the corresponding second display mode can be the supplementary display mode. Specifically, when the second display mode is the supplementary display mode, in one implementation, when playing the second information to be played, the second information to be displayed can be displayed after the second information to be played is played.

[0125] In another implementation, there may be multiple pieces of second information to be played, and correspondingly, there are multiple pieces of second information to be displayed. The second information to be played can be played in sequence. For each piece of second information to be played, after the second information to be played is finished playing, the corresponding second information to be displayed can be displayed. After the corresponding second information to be displayed is finished being displayed, the next piece of second information to be played can be played in sequence, and after the next piece of second information to be played is finished playing, the corresponding second information to be displayed for the next piece of second information to be played can be displayed until the last piece of second information to be displayed in the sequence is finished being displayed.

[0126] When the type of the second correspondence is explanatory, the corresponding second display method can be an explanatory display method. When the second display method is a supplementary display method, in one implementation, when the second information to be played is being played, the second information to be displayed can be synchronously displayed.

[0127] In another implementation, there may be multiple pieces of second information to be played, and correspondingly, there are multiple pieces of second information to be displayed. The second information to be played can be played in sequence. For each piece of second information to be played, when the second information to be played is being played, the corresponding second information to be displayed for the second information to be played can be synchronously displayed. After the corresponding second information to be displayed is finished being displayed and the second information to be played is finished playing, the next piece of second information to be played can be played in sequence, and when the next piece of second information to be played is being played, the corresponding second information to be displayed for the next piece of second information to be played can be synchronously displayed until the last piece of second information to be displayed in the sequence is finished being displayed and the last piece of second information to be played in the sequence is finished playing.

[0128] Among them, for each piece of second information to be played, the duration of playing the second information to be played and the duration of displaying the corresponding second information to be displayed can be the same or different. Specifically, it can be set according to the actual usage situation.

[0129] When the type of the second correspondence is simplified, the corresponding second display method can be a simplified display method. In this case, the second information to be displayed is text. In one implementation, when the second information to be played is being played, all the text included in the second information to be displayed can be displayed, and when the second information to be played reaches the text that belongs to the second information to be displayed, the text can be highlighted. Among them, the highlighting method can be enlarged display or can also be highlighted display.

[0130] In another implementation, when the second information to be played is being played, for each sentence included in the information to be played, when playing to that sentence, the key information (the text included in the second information to be displayed) included in that sentence can be correspondingly displayed.

[0131] It can be seen that in this embodiment, when performing playback processing on the information to be played, the corresponding display method can be adopted to perform display processing on the information to be displayed. In this way, the user can hear the content corresponding to the information to be played and see the content corresponding to the information to be displayed, improving the user experience.

[0132] Moreover, since there is a corresponding relationship between the information to be displayed and the information to be played, the two can enable the user to obtain the corresponding information visually and auditorily respectively, and can cooperate with each other to jointly complete the user's interaction requirements. Compared with the method of only playing voice, the content that the information to be displayed can provide is richer. Correspondingly, the information to be played can be simplified to a certain extent. For the user, when the information to be played is relatively concise, combined with the visual effect of the information to be displayed, the corresponding information can be obtained more intuitively and clearly.

[0133] As an implementation manner of the embodiment of the present application, the above first information to be displayed may include first address information, and the above second information to be displayed may include second address information. Among them, the first address information may be information on the storage address of the content to be displayed in the smart home appliance terminal. The second address information may be information on the storage address of the content to be displayed in the smart home appliance terminal.

[0134] The step of performing display processing on the first information to be displayed by using the first display method may include:

[0135] According to the first address information, obtain the corresponding first video from the pre-stored videos, and display the first video by using the first display method.

[0136] The step of performing display processing on the second information to be displayed by using the second display method may include:

[0137] According to the second address information, obtain the corresponding second video from the pre-stored videos, and display the second video by using the second display method.

[0138] The pre-stored videos may be videos (visible and audible materials) pre-uploaded by the user according to their actual usage requirements. It may also be videos pre-set by developers according to pre-instructions. Then, the smart home appliance terminal can obtain the videos set by the developers from the server and store them.

[0139] It can be seen that in this embodiment, compared with the method where the information to be displayed is a video, the server can send the information to be displayed to the smart home appliance terminal more quickly and can save network resources.

[0140] As an implementation manner of an embodiment of the present application, after the step of obtaining the control information corresponding to the control instruction, the method may further include:

[0141] According to the control information corresponding to the control instruction and the pre-stored corresponding relationship between the control information and the usage information, determine the corresponding usage information as the third information to be used, where the third information to be used includes third information to be displayed and third information to be played, and there is a corresponding relationship between the third information to be displayed and the third information to be played;

[0142] Process the third information to be displayed and the third information to be played.

[0143] For some common control instructions, the corresponding relationship between the control information and the usage information may be pre-stored in the intelligent home appliance terminal. For example, the intelligent home appliance terminal may store the corresponding relationship between the image of the gesture for adjusting the volume, the animation for volume adjustment, and the voice for volume adjustment.

[0144] The step of processing the third information to be displayed and the third information to be played is the same as the step of processing the first information to be displayed and the first information to be played. For specific descriptions, reference may be made to the description of the step of processing the first information to be displayed and the first information to be played, which will not be elaborated here.

[0145] It can be seen that in this embodiment, when the intelligent home appliance terminal is offline, it can determine the third information to be used according to the pre-stored corresponding relationship between the control information and the usage information, so as to process the third information to be displayed and the third information to be played subsequently. In this way, the user can intuitively obtain the information corresponding to the control instruction from both the visual and auditory aspects, which can improve the user experience.

[0146] As an implementation manner of an embodiment of the present application, the display screen of the intelligent home appliance terminal may display the information to be displayed in a variety of different forms, where the information to be displayed may include the first information to be displayed, the second information to be displayed, and the third information to be displayed.

[0147] In the first form, the intelligent home appliance terminal may display the information to be displayed in the form of an assistant dialog box. The user can drag the assistant dialog box by means of touch, click, etc., can wake up the assistant dialog box through a voice control instruction, and can also close the assistant dialog box through a voice control instruction. The information to be displayed may be any one of pictures, videos, and texts.

[0148] In the second form, the intelligent home appliance terminal may display the video or image corresponding to the gesture control instruction in a gesture display window. The gesture display window may be located above other applications displayed on the display screen.

[0149] In the third form, when the information to be displayed is an animation, the smart home appliance terminal can display the animation using the animation center window. The animation center window can be located above other applications displayed on the display screen.

[0150] In the fourth form, when the information to be displayed is a card-type picture, the smart home appliance terminal can display the card-type picture in the form of a pop-up window.

[0151] It can be seen that in this embodiment, different forms can be used to display the information to be displayed, improving the user's visual experience.

[0152] As an implementation manner of the embodiments of the present application, the smart home appliance terminal may store a first multimodal interaction framework, and / or the server may store a second multimodal interaction framework. Among them, the first multimodal interaction framework may be provided with control information corresponding to some common control instructions.

[0153] The second multimodal interaction framework may include information on all functions corresponding to the smart home appliance terminal. For example, the second multimodal interaction framework may include options corresponding to application programs installed on the smart home appliance terminal and the application terminal. Specifically, the trigger information corresponding to different trigger buttons may be set in the second multimodal interaction framework.

[0154] For the smart home appliance terminal, it can be determined whether the control information hits the first multimodal interaction framework. When the control information hits the first multimodal interaction framework, the steps of determining the corresponding usage information according to the control information corresponding to the control instruction and the pre-stored correspondence between the control information and the usage information, and using the corresponding usage information as the third information to be used can be executed.

[0155] When the control information does not hit the first multimodal interaction framework, the voice information and display information indicating the failure of the execution of the pre-stored control instruction can be played. When the smart home appliance terminal is online, the control information can also be sent to the server, so as to perform the subsequent steps of determining the information to be used.

[0156] For the server, when the intent information included in the recognition result hits the second multimodal interaction framework, the first information to be used can be correspondingly determined. After the first information to be used is sent to the smart home appliance terminal, the smart home appliance terminal can process the first information to be displayed and the first information to be played.

[0157] When the first information to be used includes control parameters, the control parameters can be executed, and a return value can be generated according to the execution situation, and the return value is sent to the server. When the server receives the return value and the return value hits the second multimodal interaction framework, the second information to be used can be determined and sent to the smart home appliance terminal, and the smart home appliance terminal can then process the second information to be displayed and the second information to be played.

[0158] To more conveniently understand an intelligent home appliance terminal control method combining vision and audition provided by an embodiment of the present application, the following will introduce an intelligent home appliance terminal control method combining vision and audition provided by an embodiment of the present application with reference to Figure 3 an intelligent home appliance terminal control method combining vision and audition provided by an embodiment of the present application will be introduced.

[0159] As Figure 3 shown, the user can issue a voice for uploading visible and speakable materials, and the intelligent home appliance terminal can recognize the voice for uploading visible and speakable materials to obtain location information, and the intelligent home appliance terminal can download and store the visible and speakable materials according to the location information.

[0160] Furthermore, the user can issue a voice control instruction. After receiving the voice control instruction, the intelligent home appliance terminal (not shown in the figure) can convert the voice control instruction into text information and send the text information to the server (not shown in the figure), and the server can perform intention recognition on the text information.

[0161] Specifically, the text information can be added to an intention recognition queue to implement intention recognition, obtain a recognition result, determine the first information to be used according to the recognition result, and send the first information to be used to the intelligent home appliance terminal.

[0162] When the first information to be used includes the first information to be displayed and the first information to be played, the intelligent home appliance terminal can process the first information to be displayed and the first information to be played.

[0163] The following describes a second intelligent home appliance terminal control method combining vision and audition provided by the present application. The second intelligent home appliance terminal control method combining vision and audition described below can be mutually referred to the first intelligent home appliance terminal control method combining vision and audition described above.

[0164] As Figure 4 shown, an embodiment of the present application provides a second intelligent home appliance terminal control method combining vision and audition. The method is applied to a server, and the method includes:

[0165] S401, receiving control information sent by an intelligent home appliance terminal.

[0166] Wherein, the control information is the information corresponding to the control instruction obtained by the intelligent home appliance terminal after obtaining the control instruction.

[0167] S402. Perform intention recognition on the control information to obtain a recognition result, and determine first information to be used according to the recognition result.

[0168] S403. Send the first information to be used to the intelligent home appliance terminal, so that when the first information to be used includes first information to be displayed and first information to be played, the intelligent home appliance terminal processes the first information to be displayed and the first information to be played.

[0169] Wherein, there is a first corresponding relationship between the first information to be displayed and the first information to be played.

[0170] It can be seen that in this embodiment, the intelligent home appliance terminal can process the first information to be displayed and the first information to be played. For the user, after sending a control instruction, the user can see the first information to be displayed on the intelligent home appliance terminal and hear the first information to be played, that is, the user can see the visual information corresponding to the control instruction displayed on the intelligent home appliance terminal and can hear the auditory information corresponding to the control instruction played by the intelligent home appliance. The user can intuitively obtain the information corresponding to the control instruction from both visual and auditory aspects, which can improve the user experience.

[0171] As an implementation manner of the embodiment of the present application, after the step of sending the first information to be used to the intelligent home appliance terminal, the method further includes:

[0172] Receive a return value sent by the intelligent home appliance terminal, where the return value is a value generated by the intelligent home appliance terminal according to the execution situation after executing the control parameter when the first information to be used includes a control parameter.

[0173] Determine second information to be used based on the return value, and send the second information to be used to the intelligent home appliance terminal, so that when the second information to be used includes second information to be displayed and second information to be played, the intelligent home appliance terminal processes the second information to be displayed and the second information to be played, where there is a second corresponding relationship between the second information to be displayed and the second information to be played.

[0174] Next, a first audio-visual combined intelligent home appliance terminal control device provided by the present application will be described. The first audio-visual combined intelligent home appliance terminal control device described below can be correspondingly referred to the first audio-visual combined intelligent home appliance terminal control method described above.

[0175] As Figure 5 shown, the embodiment of the present application provides a first audio-visual combined intelligent home appliance terminal control device. The device is applied to an intelligent home appliance terminal, and the device includes:

[0176] A first acquisition module 510, configured to acquire a control instruction.

[0177] A second acquisition module 520, configured to acquire control information corresponding to the control instruction, and send the control information to a server, so that the server performs intention recognition on the control information to obtain a recognition result, and determines first information to be used according to the recognition result.

[0178] A first receiving module 530, configured to receive the first information to be used sent by the server.

[0179] A first processing module 540, configured to process the first information to be displayed and the first information to be played when the first information to be used includes the first information to be displayed and the first information to be played.

[0180] Wherein, there is a first corresponding relationship between the first information to be displayed and the first information to be played.

[0181] As an implementation manner of an embodiment of the present application, the above device may further include:

[0182] An execution module, configured to execute the control parameter when the first information to be used includes a control parameter after receiving the first information to be used sent by the server, and generate a return value according to the execution situation.

[0183] A return value sending module, configured to send the return value to the server, so that the server determines second information to be used based on the return value.

[0184] A third receiving module, configured to receive the second information to be used sent by the server.

[0185] A second processing module, configured to process the second information to be displayed and the second information to be played when the second information to be used includes the second information to be displayed and the second information to be played.

[0186] Wherein, there is a second corresponding relationship between the second information to be displayed and the second information to be played.

[0187] As an implementation manner of an embodiment of the present application, the above first processing module 540 may include:

[0188] A first determination unit, configured to determine a corresponding first display manner according to the type of the first corresponding relationship.

[0189] A first processing unit, configured to perform a display process on the first information to be displayed by using the first display manner when performing a playback process on the first information to be played.

[0190] The above-mentioned second processing module may include:

[0191] A second determination unit, configured to determine a corresponding second display manner according to the type of the second correspondence relationship.

[0192] A second processing unit, configured to, when performing playback processing on the second to-be-played information, perform display processing on the second to-be-displayed information by using the second display manner.

[0193] As an implementation manner of an embodiment of the present application, the above-mentioned first to-be-displayed information may include first address information, and the above-mentioned second to-be-displayed information may include second address information.

[0194] The above-mentioned first processing unit is specifically configured to obtain a corresponding first video from a pre-stored video according to the first address information, and display the first video by using the first display manner.

[0195] The above-mentioned second processing unit is specifically configured to obtain a corresponding second video from a pre-stored video according to the second address information, and display the second video by using the second display manner.

[0196] As an implementation manner of an embodiment of the present application, the above-mentioned device may further include:

[0197] A determination module, configured to, after obtaining control information corresponding to the control instruction, determine corresponding usage information according to the control information corresponding to the control instruction and a pre-stored correspondence relationship between control information and usage information, and use the corresponding usage information as third to-be-used information.

[0198] Wherein, the third to-be-used information includes third to-be-displayed information and third to-be-played information, and there is a correspondence relationship between the third to-be-displayed information and the third to-be-played information.

[0199] A third processing module, configured to process the third to-be-displayed information and the third to-be-played information.

[0200] Next, a second audio-visual combined intelligent home appliance terminal control device provided by the present application is described. The second audio-visual combined intelligent home appliance terminal control device described below can be correspondingly referred to the second audio-visual combined intelligent home appliance terminal control method described above.

[0201] As Figure 6 shown, an embodiment of the present application provides a second audio-visual combined intelligent home appliance terminal control device. The device is applied to a server, and the device includes:

[0202] A second receiving module 610, configured to receive control information sent by the intelligent home appliance terminal.

[0203] Wherein, the control information is information corresponding to the control instruction obtained by the intelligent home appliance terminal after obtaining the control instruction;

[0204] An identification module 620, configured to perform intention identification on the control information to obtain an identification result, and determine first information to be used according to the identification result;

[0205] A sending module 630, configured to send the first information to be used to the intelligent home appliance terminal, so that when the first information to be used includes first information to be displayed and first information to be played, the intelligent home appliance terminal processes the first information to be displayed and the first information to be played.

[0206] Wherein, there is a first corresponding relationship between the first information to be displayed and the first information to be played.

[0207] As an implementation manner of an embodiment of the present application, the above device may further include:

[0208] A fourth receiving module, configured to receive a return value sent by the intelligent home appliance terminal after sending the first information to be used to the intelligent home appliance terminal.

[0209] Wherein, the return value is a value generated by the intelligent home appliance terminal according to an execution situation after executing the control parameter when the first information to be used includes a control parameter.

[0210] An information sending module, configured to determine second information to be used based on the return value, and send the second information to be used to the intelligent home appliance terminal, so that when the second information to be used includes second information to be displayed and second information to be played, the intelligent home appliance terminal processes the second information to be displayed and the second information to be played.

[0211] Wherein, there is a second corresponding relationship between the second information to be displayed and the second information to be played.

[0212] Figure 7 Schematically illustrates the physical structure of an electronic device, such as Figure 7As shown, the electronic device can be a smart home appliance terminal or a server. The electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logical instructions in the memory 730 to execute the control method for the smart home appliance terminal with audio-visual combination.

[0213] In addition, when the logical instructions in the above-mentioned memory 730 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0214] On the other hand, this application also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute what is provided by the above-mentioned various methods to execute the control method for the smart home appliance terminal with audio-visual combination.

[0215] On yet another hand, this application also provides a computer-readable storage medium. The computer-readable storage medium includes a stored program. Among them, when the program runs, it executes what is provided by the above-mentioned various methods to execute the control method for the smart home appliance terminal with audio-visual combination.

[0216] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0217] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent home appliance terminal control method combining audio and visual, characterized in that, the method is applied to an intelligent home appliance terminal, and the method includes: Obtaining a control instruction; Obtaining control information corresponding to the control instruction, and sending the control information to a server, so that the server performs intent recognition on the control information to obtain a recognition result, and determines first information to be used according to the recognition result; Receiving the first information to be used sent by the server; When the first information to be used includes first information to be displayed and first information to be played, processing the first information to be displayed and the first information to be played, wherein there is a first corresponding relationship between the first information to be displayed and the first information to be played; Wherein, the type of the first corresponding relationship includes one of supplementary, explanatory, and simplified. The meaning of the supplementary type is that the first information to be played and the first information to be displayed complement each other. The meaning of the explanatory type is that the first information to be played and the first information to be displayed explain each other. The meaning of the simplified type is that the first information to be displayed is a simplification of the first information to be played.

2. The intelligent home appliance terminal control method combining audio and visual according to claim 1, characterized in that, after the step of receiving the first information to be used sent by the server, the method further includes: When the first information to be used includes control parameters, executing the control parameters and generating a return value according to the execution situation; Sending the return value to the server, so that the server determines second information to be used based on the return value; Receiving the second information to be used sent by the server; When the second information to be used includes second information to be displayed and second information to be played, processing the second information to be displayed and the second information to be played, wherein there is a second corresponding relationship between the second information to be displayed and the second information to be played.

3. The intelligent home appliance terminal control method combining audio and visual according to claim 2, characterized in that, The step of processing the first information to be displayed and the first information to be played includes: Determining a corresponding first display method according to the type of the first corresponding relationship; When performing playback processing on the first information to be played, performing display processing on the first information to be displayed by using the first display method; The step of processing the second information to be displayed and the second information to be played includes: Determining a corresponding second display method according to the type of the second corresponding relationship; When performing playback processing on the second information to be played, performing display processing on the second information to be displayed by using the second display method.

4. The intelligent home appliance terminal control method combining audio and visual according to claim 3, characterized in that, The first information to be displayed includes first address information, and the second information to be displayed includes second address information; The step of performing display processing on the first information to be displayed by using the first display method includes: According to the first address information, obtain the corresponding first video from the pre-stored videos; Display the first video in the first display mode; The step of performing display processing on the second information to be displayed in the second display mode includes: According to the second address information, obtain the corresponding second video from the pre-stored videos; Display the second video in the second display mode.

5. The method for controlling an intelligent home appliance terminal with audio-visual combination according to any one of claims 1-4, characterized in that, After the step of obtaining the control information corresponding to the control instruction, the method further includes: According to the control information corresponding to the control instruction and the pre-stored corresponding relationship between the control information and the usage information, determine the corresponding usage information, and use the corresponding usage information as the third information to be used, where the third information to be used includes the third information to be displayed and the third information to be played, and there is a corresponding relationship between the third information to be displayed and the third information to be played; Process the third information to be displayed and the third information to be played.

6. A method for controlling an intelligent home appliance terminal with audio-visual combination, characterized in that, The method is applied to a server, and the method includes: Receive the control information sent by the intelligent home appliance terminal, where the control information is the information corresponding to the control instruction obtained by the intelligent home appliance terminal after obtaining the control instruction; Perform intention recognition on the control information to obtain a recognition result, and determine the first information to be used according to the recognition result; Send the first information to be used to the intelligent home appliance terminal, so that when the first information to be used includes the first information to be displayed and the first information to be played, the intelligent home appliance terminal processes the first information to be displayed and the first information to be played, where there is a first corresponding relationship between the first information to be displayed and the first information to be played; Wherein, the type of the first corresponding relationship includes one of supplementary type, explanatory type and simplified type. The meaning of the supplementary type is that the first information to be played and the first information to be displayed complement each other. The meaning of the explanatory type is that the first information to be played and the first information to be displayed explain each other. The meaning of the simplified type is that the first information to be displayed is a simplification of the first information to be played.

7. The method for controlling an intelligent home appliance terminal with audio-visual combination according to claim 6, characterized in that, After the step of sending the first information to be used to the intelligent home appliance terminal, the method further includes: Receive the return value sent by the intelligent home appliance terminal, where the return value is a value generated by the intelligent home appliance terminal according to the execution situation after executing the control parameter when the first information to be used includes the control parameter. Determine the second information to be used based on the return value, and send the second information to be used to the intelligent home appliance terminal, so that the intelligent home appliance terminal processes the second information to be displayed and the second information to be played when the second information to be used includes the second information to be displayed and the second information to be played, where there is a second corresponding relationship between the second information to be displayed and the second information to be played.

8. An intelligent home appliance terminal control device combining vision and audition, characterized in that the device is applied to an intelligent home appliance terminal, and the device includes: A first acquisition module, configured to acquire a control instruction; A second acquisition module, configured to acquire the control information corresponding to the control instruction, and send the control information to the server, so that the server performs intention recognition on the control information, obtains an identification result, and determines the first information to be used according to the identification result; A first receiving module, configured to receive the first information to be used sent by the server; A first processing module, configured to process the first information to be displayed and the first information to be played when the first information to be used includes the first information to be displayed and the first information to be played, where there is a first corresponding relationship between the first information to be displayed and the first information to be played; wherein, the type of the first corresponding relationship includes one of supplementary type, explanatory type and simplified type. The meaning of the supplementary type is that the first information to be played and the first information to be displayed complement each other. The meaning of the explanatory type is that the first information to be played and the first information to be displayed explain each other. The meaning of the simplified type is that the first information to be displayed is a simplification of the first information to be played.

9. An intelligent home appliance terminal control device combining vision and audition, characterized in that the device is applied to a server, and the device includes: A second receiving module, configured to receive the control information sent by the intelligent home appliance terminal, where the control information is the information corresponding to the control instruction acquired by the intelligent home appliance terminal after acquiring the control instruction; An identification module, configured to perform intention recognition on the control information, obtain an identification result, and determine the first information to be used according to the identification result; A sending module, configured to send the first information to be used to the intelligent home appliance terminal, so that the intelligent home appliance terminal processes the first information to be displayed and the first information to be played when the first information to be used includes the first information to be displayed and the first information to be played, where there is a first corresponding relationship between the first information to be displayed and the first information to be played. Among them, the type of the first corresponding relationship includes one of supplementary type, explanatory type, and simplified type. The meaning of the supplementary type is that the first information to be played and the first information to be displayed complement each other. The meaning of the explanatory type is that the first information to be played and the first information to be displayed explain each other. The meaning of the simplified type is that the first information to be displayed is a simplification of the first information to be played.

10. A computer-readable storage medium, characterized in that, the computer-readable storage medium includes a stored program, wherein when the program runs, it executes the method according to any one of claims 1 to 5, or when the program runs, it executes the method according to claim 6 or 7.

11. An electronic device, comprising a memory and a processor, characterized in that, a computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 5 through the computer program, or the processor is configured to execute the method according to claim 6 or 7 through the computer program.

Images