Multi-modal interaction method of intelligent equipment, storage medium and electronic device

Abstract

The invention discloses a multi-mode interaction method of intelligent equipment, a storage medium and an electronic device, and relates to the technical field of intelligent home furnishing.The multi-mode interaction method of the intelligent equipment comprises the steps that knocking information sent by the intelligent equipment is received, the knock information is obtained by a voice module in the intelligent device to obtain interaction information generated when a user knocks the intelligent device, sound analysis is performed on the knock information, a first confidence coefficient of the knock information is determined, the first confidence coefficient represents the degree of the knock information used for triggering the control instruction, and when the first confidence coefficient is greater than a preset threshold value, the knock information is sent to the intelligent device. Whether the knocking function of the intelligent equipment is started or not is judged, if the knocking function of the intelligent equipment is started, and the intelligent equipment is in a standby state, a target control instruction corresponding to the knocking function is inquired, and the target control instruction is sent to the intelligent equipment; the method improves the control convenience of the intelligent device, ensures the operation accuracy, and improves the user experience.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a multimodal interaction method, storage medium, and electronic device for smart devices. Background Technology

[0002] With the rapid development of smart device technology, smart devices, as indispensable electrical appliances in daily family life, are constantly updating their interaction methods.

[0003] There are many ways to interact with existing smart devices, but they generally rely on traditional user operation modes, such as users performing complex multi-step manual settings on the smart device panel, adjusting parameters remotely through mobile applications, and using smart voice assistants to issue voice commands to achieve multimodal interaction with smart devices.

[0004] However, existing technologies suffer from problems such as complex and lengthy operation processes for smart devices and low accuracy of speech recognition in noisy environments. Summary of the Invention

[0005] This application provides a multimodal interaction method, storage medium, and electronic device for smart devices, in order to solve the problems of complex and lengthy operation processes of smart devices and low accuracy of speech recognition in noisy environments.

[0006] Firstly, this application provides a multimodal interaction method for smart devices, applied to a cloud server, comprising:

[0007] The device receives tapping information sent by the smart device, wherein the tapping information is obtained by the voice module in the smart device from the interactive information generated by the user tapping the smart device;

[0008] The tapping information is analyzed by sound to determine a first confidence level of the tapping information, wherein the first confidence level characterizes the degree to which the tapping information is used to trigger control commands;

[0009] If the first confidence level is greater than a preset threshold, it is determined whether the tapping function of the smart device is enabled. If the tapping function of the smart device is enabled and the smart device is in standby mode, the target control command corresponding to the tapping function is queried and the target control command is sent to the smart device.

[0010] Optional, also includes:

[0011] If the tapping function of the smart device is not enabled and the smart device is in standby mode, a first prompt message is sent to the smart device. The first prompt message is used to prompt the user whether to enable the tapping function.

[0012] Optional, also includes:

[0013] Receive interactive voice messages sent by the smart device and parse the interactive voice messages;

[0014] If the interactive voice instruction enables the tapping function, then the tapping function of the smart device is enabled, and a second prompt message is sent to the smart device. The second prompt message is used to prompt one or more of the following: the tapping function is enabled, the tapping gesture, and the change method of the control command corresponding to the tapping function.

[0015] Optional, also includes:

[0016] Determine whether this is the first use of the tap function after it has been enabled;

[0017] If it is the first time using the device, a third prompt message is sent to the smart device. The third prompt message is used to prompt one or more of the following: the target control command has been started, and the method of changing the control command corresponding to the tapping function.

[0018] If this is not the first time using the device, a fourth prompt message is sent to the smart device, which indicates that the target control command has been started.

[0019] Optional, also includes:

[0020] If the tapping function of the smart device is enabled and the smart device is in working state, a fifth prompt message is sent to the smart device. The fifth prompt message is used to prompt the user to control the smart device through the tapping function when the smart device is in standby state.

[0021] Secondly, this application provides a multimodal interaction method for intelligent devices, applied to intelligent control systems, including:

[0022] In response to a tapping signal triggered by a user tapping the smart device, a second confidence level of the tapping signal is identified, wherein the second confidence level characterizes the degree to which the tapping signal is used to trigger a control command;

[0023] If the second confidence level is greater than a preset threshold, the tapping information is sent to the cloud server;

[0024] The system receives target control instructions sent by the cloud server, which are used to control the operation of the smart device.

[0025] Optional, also includes:

[0026] Receive user's interactive voice and send the interactive voice to the cloud server;

[0027] The system receives a second prompt message sent by the cloud server. The second prompt message is used to indicate one or more of the following: the tapping function has been enabled, the tapping gesture, and the method of changing the control command corresponding to the tapping function.

[0028] Optional, also includes:

[0029] After the smart device is powered on, if the tapping function is not enabled, a sixth prompt message is output, which is used to prompt the method of enabling the tapping function; if the tapping function is enabled, but the tapping function is not used within a preset time, a seventh prompt message is output, which is used to prompt the gesture for the tapping function.

[0030] Thirdly, this application provides a multimodal interaction device for a smart device, comprising:

[0031] A receiving module is used to receive tapping information sent by the smart device, wherein the tapping information is obtained by the voice module in the smart device from the interactive information generated by the user tapping the smart device;

[0032] A determination module is used to perform sound analysis on the tapping information and determine a first confidence level of the tapping information, wherein the first confidence level characterizes the degree to which the tapping information is used to trigger a control command;

[0033] The judgment module is used to determine whether the tapping function of the smart device is enabled when the first confidence level is greater than a preset threshold.

[0034] The sending module is used to query the target control command corresponding to the tapping function and send the target control command to the smart device if the tapping function of the smart device is enabled and the smart device is in standby mode.

[0035] Optionally, the sending module is further configured to send a first prompt message to the smart device if the tapping function of the smart device is not enabled and the smart device is in standby mode, wherein the first prompt message is used to prompt the user whether to enable the tapping function.

[0036] Optionally, the receiving module is further configured to receive interactive voice sent by the smart device and parse the interactive voice;

[0037] The sending module is further configured to enable the tapping function of the smart device if the interactive voice instruction enables the tapping function, and send a second prompt message to the smart device. The second prompt message is used to prompt one or more of the following: the tapping function has been enabled, the gesture of the tapping function, and the change method of the control command corresponding to the tapping function.

[0038] Optionally, the judgment module is also used to determine whether the current use of the tapping function is the first use after the tapping function has been enabled;

[0039] The sending module is also configured to send a third prompt message to the smart device if it is the first time it is used. The third prompt message is used to prompt one or more of the following: the target control command has been started, and the change method of the control command corresponding to the tapping function.

[0040] The sending module is further configured to send a fourth prompt message to the smart device if it is not the first time it is used, the fourth prompt message being used to indicate that the target control command has been started.

[0041] Optionally, the sending module is further configured to send a fifth prompt message to the smart device if the tapping function of the smart device is enabled and the smart device is in working state. The fifth prompt message is used to prompt the user to control the smart device through the tapping function when the smart device is in standby state.

[0042] Fourthly, this application provides a multimodal interaction device for smart devices, comprising:

[0043] Memory;

[0044] processor;

[0045] The memory stores computer-executed instructions;

[0046] The processor executes computer execution instructions stored in the memory to implement the multimodal interaction method of the intelligent device as described in the first aspect and various possibilities of the first aspect, as well as the second aspect and various possibilities of the second aspect.

[0047] Fifthly, this application provides a computer storage medium, characterized in that the computer storage medium stores computer execution instructions, which, when executed by a processor, are used to implement the multimodal interaction method of the intelligent device as described in the first aspect and various possibilities of the first aspect, as well as the second aspect and various possibilities of the second aspect.

[0048] The multimodal interaction method for smart devices provided in this application receives tapping information sent by the smart device. This tapping information is obtained by the voice module in the smart device from the user's tapping of the smart device. The method performs sound analysis on the tapping information to determine a first confidence level, which characterizes the degree to which the tapping information can trigger a control command. If the first confidence level is greater than a preset threshold, it determines whether the tapping function of the smart device is enabled. If the tapping function is enabled and the smart device is in standby mode, it queries the target control command corresponding to the tapping function and sends the target control command to the smart device. This method improves the convenience of controlling smart devices while ensuring operational accuracy and enhancing the user experience. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is an interaction diagram of the smart device provided in this application;

[0052] Figure 2 This is the flow chart of the multimodal interaction method for smart devices provided in this application. Figure 1 ;

[0053] Figure 3 This is the flow chart of the multimodal interaction method for smart devices provided in this application. Figure 2 ;

[0054] Figure 4 This is the flow chart of the multimodal interaction method for smart devices provided in this application. Figure 3 ;

[0055] Figure 5 This is a schematic diagram of the structure of the multimodal interaction device for the intelligent device provided in this application;

[0056] Figure 6 This is a structural schematic diagram of the multimodal interaction device of the smart device provided in this application. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0058] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0059] According to one aspect of the embodiments of this application, an interaction method for a smart device is provided. This interaction method for a smart device is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned interaction method for a smart device can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0060] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein.

[0063] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0064] With the rapid development of smart home appliance technology, smart devices have become deeply integrated into and simplified our daily family life, and their interaction methods are constantly evolving and innovating to provide a more convenient and intelligent user experience.

[0065] While existing smart devices offer a wide variety of interaction methods, they are generally still limited by traditional user interaction paradigms: users either need to complete tedious manual settings step by step on the smart device's own control panel; or rely on mobile applications to remotely adjust parameters from a distance; or use voice commands issued by a smart voice assistant to control the smart device.

[0066] However, existing technologies suffer from problems such as complex and lengthy operation processes for smart devices and low accuracy of voice recognition in noisy environments, resulting in low efficiency and poor user experience.

[0067] To address the aforementioned issues, this application provides a multimodal interaction method for smart devices. This method utilizes the smart device's built-in voice module to convert the physical vibrations generated by a user tapping the device's surface into processable sound information. By performing detailed sound analysis on this tapping information, the confidence level of the tapping intention is determined. When the confidence level exceeds a preset threshold, and after confirming that the tapping function is enabled and the smart device is in standby mode, a control command matching the tapping mode is automatically executed. This method not only simplifies the operation process of controlling smart devices and ensures operational accuracy, but also effectively improves the user experience.

[0068] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0069] Figure 2 This is the flow chart of the multimodal interaction method for smart devices provided in this application. Figure 1 The execution entity in this embodiment is, for example, a cloud server. Figure 2 As shown, the multimodal interaction method for smart devices provided in this embodiment includes:

[0070] S201: Receive tapping information sent by the smart device. The tapping information is obtained by the voice module in the smart device from the interactive information generated by the user tapping the smart device.

[0071] First, the smart device is equipped with a highly sensitive tap sensor and a corresponding signal transmission module. The tap sensor can capture the user's tapping motion and convert it into a recognizable electrical signal. Simultaneously, the smart device needs to be integrated with a cloud server to ensure that the tapping information can be successfully transmitted to the cloud server.

[0072] Among the diverse methods for controlling smart devices, tapping stands out due to its unique physical interaction characteristics. This method utilizes specific vibrations or sound waves generated by tapping as control commands. Compared to media such as sound or light, which are easily affected by environmental interference, it effectively reduces noise interference from the surrounding environment. Because tapping behavior often has clear spatiotemporal characteristics and high recognizability, it ensures accurate transmission and recognition of control commands even in noisy environments. This significantly improves the accuracy and reliability of interaction between users and smart devices, bringing users a smoother and smarter home living experience.

[0073] S202: Perform sound analysis on the tapping information to determine the first confidence level of the tapping information. The first confidence level represents the degree to which the tapping information can trigger control commands.

[0074] Among these, sound analysis of tapping information refers to the process of analyzing and processing non-traditional speech signals generated by tapping actions (physical tapping by smart devices) using natural language processing or speech recognition technologies. The cloud server converts the tapping sequence, rhythm, force, and other features into a computer-understandable format and attempts to extract meaningful information or instructions from them.

[0075] Determining the first confidence level of the tap information is a quantitative assessment of the parsing results. It reflects the reliability of the cloud server's belief that the tap information is indeed used to trigger a certain control command.

[0076] Understandably, the first step involves preprocessing the tapping data, such as cleaning and denoising, to extract useful features for analysis and recognition, such as tapping sequence patterns, rhythm variations, and force distribution. These features should effectively characterize the user's tapping intentions and characteristics. Machine learning or deep learning algorithms, such as Hidden Markov Models (HMMs) and Recurrent Neural Networks (RNNs), are then used to train a large amount of labeled tapping data, establishing a mapping relationship between tapping features and control commands. The cloud server receives the user's tapping information in real time, analyzes it using the trained model, and compares the analysis results with historical tapping data to determine the initial confidence level. Based on the confidence level, the cloud server decides whether to execute the corresponding control command.

[0077] S203: If the first confidence level is greater than a preset threshold, determine whether the tapping function of the smart device is enabled. If the tapping function of the smart device is enabled and the smart device is in standby mode, query the target control command corresponding to the tapping function and send the target control command to the smart device.

[0078] Among them, when the first confidence level is greater than a preset threshold, the state of the tapping function is determined. The cloud server can more accurately identify the user's true intention and reduce the possibility of misoperation.

[0079] Understandably, firstly, advanced signal processing and recognition technologies need to be integrated into the cloud server to accurately calculate the first confidence level of the tapping operation. Simultaneously, the cloud server needs to be able to acquire the smart device's status information in real time, including whether the tapping function is enabled and whether it is currently in standby mode. When the first confidence level exceeds a preset threshold, the cloud server will execute corresponding judgment logic based on the smart device's status information. When the smart device's tapping function is enabled and the smart device is in standby mode, if a tapping action is detected, the cloud server accurately identifies the occurrence of the tapping event and its characteristics (such as tapping location, force, and number of taps) through sensor data analysis and algorithm processing. Based on the identified tapping characteristics, the cloud server queries a preset mapping relationship to find the corresponding target control command. Then, it sends the target control command to the smart device via a communication protocol. After receiving the target control command, the smart device executes the corresponding operation and may provide feedback to the user through lights, sounds, or screen displays.

[0080] The multimodal interaction method for smart devices provided in this embodiment receives tapping information sent by the smart device. This tapping information is obtained by the voice module within the smart device, which processes the user's tapping of the device. The tapping information is then analyzed to determine a first confidence level, which represents the degree to which the tapping information can trigger a control command. If the first confidence level is greater than a preset threshold, it is determined whether the smart device's tapping function is enabled. If the tapping function is enabled and the smart device is in standby mode, the target control command corresponding to the tapping function is queried and sent to the smart device. This method improves the convenience of controlling the smart device while ensuring operational accuracy, thus enhancing the user experience.

[0081] Figure 3 This is the flow chart of the multimodal interaction method for smart devices provided in this application. Figure 2 .like Figure 3 As shown, in this embodiment... Figure 2 Based on the embodiments, the multimodal interaction method of smart devices is described in detail. The multimodal interaction method of smart devices shown in this embodiment includes:

[0082] S301: Receives tapping information sent by the smart device. The tapping information is obtained by the voice module in the smart device from the interactive information generated by the user tapping the smart device.

[0083] Step S301 is similar to step S201, and will not be described again here.

[0084] S302: Perform sound analysis on the tapping information to determine the first confidence level of the tapping information. The first confidence level represents the degree to which the tapping information can trigger control commands.

[0085] Step S202 is similar to step S102, and will not be described again here.

[0086] S303: If the first confidence level is greater than a preset threshold, determine whether the tapping function of the smart device is enabled. If yes, proceed to step S207 or step S211; if no, proceed to step S204.

[0087] The preset threshold can be either 0.9 or 0.92. This application does not impose specific restrictions on the preset threshold.

[0088] Understandably, when the first confidence level exceeds the preset threshold, the cloud server will automatically check and determine whether the tap control function of the current smart device has been activated or enabled by the user, so as to ensure that subsequent control commands can be executed based on the user's true intention and the device's functional status.

[0089] S304: If the smart device is in standby mode, a first prompt message is sent to the smart device. The first prompt message is used to prompt the user whether to enable the tapping function.

[0090] The standby state is used to indicate that the smart device is not performing its functions.

[0091] When the tap function of a smart device is not enabled and the smart device is in standby mode, the cloud server will proactively send a first prompt message to the smart device. The first prompt message is intended to ask or remind the user whether they want to enable the tap function so that they can control the smart device through tapping.

[0092] For example, the first prompt message could be, "The tap-to-start function is not yet enabled. Do you want me to enable it for you?"

[0093] S305: Receives interactive voice messages sent by smart devices and parses the interactive voice messages.

[0094] When a smart device issues its first prompt to guide the user's interaction, the user will respond by engaging in a conversation with the device via voice. At this point, the smart device will send the collected voice messages to a cloud server. The cloud server will then initiate a parsing process to decode, recognize, and semantically analyze the received voice messages to accurately understand the user's intentions and commands.

[0095] S306: If the interactive voice instruction enables the tapping function, then the tapping function of the smart device is enabled, and a second prompt message is sent to the smart device. The second prompt message is used to prompt one or more of the following: the tapping function is enabled, the gesture of the tapping function, and the change method of the control command corresponding to the tapping function.

[0096] The cloud server first uses voice recognition technology to analyze the user's voice interaction and identify the commands. Once the command is confirmed to be activating the tap function on the smart device, the cloud server sends a command to the smart device to activate the tap function via the control interface. Subsequently, the cloud server generates a second notification message containing necessary information, such as confirmation that the tap function is enabled, instructions on using the tap function's gestures, and instructions on how to change control commands. Finally, the cloud server sends the second notification message to the user via the smart device's display screen, voice assistant, or other communication methods, ensuring the user receives this information promptly.

[0097] For example, the second prompt message could be "The tap-to-start function is now on! Try tapping it twice. You can change the startup program in the smart device's smart assistant!" Here, "The tap-to-start function is now on" indicates that the tapping function is enabled, the tapping gesture is "Try tapping it twice," and the way to change the control command corresponding to the tapping function is "You can change the startup program in the smart device's smart assistant."

[0098] S307: If the smart device is in standby mode, query the target control command corresponding to the tapping function and send the target control command to the smart device.

[0099] When the tapping function of a smart device is enabled and the device is in standby mode, if a tapping action is detected, the cloud server accurately identifies the occurrence and characteristics of the tapping event (such as tapping location, force, and number of taps) through sensor data analysis and algorithm processing. Based on the identified tapping characteristics, the cloud server queries a preset mapping relationship to find the corresponding target control command. Then, it sends the target control command to the smart device via a communication protocol. After receiving the target control command, the smart device executes the corresponding operation and may provide feedback to the user through lights, sounds, or screen displays.

[0100] S308: Determine whether this is the first use of the tap function after it has been enabled. If yes, proceed to step S210; otherwise, proceed to step S209.

[0101] After a user successfully enables the tap function on their smart device, the cloud server needs to identify and confirm whether the tapping action is the first time the function has been enabled. Providing detailed operation guides, tutorials, or safety tips is essential for new users or those using the tap function for the first time. This helps users quickly familiarize themselves with the device and avoid accidental operations. By recognizing first-time tapping, the cloud server can automatically trigger these guidance messages, improving the user experience.

[0102] S309: Send a fourth prompt message to the smart device. The fourth prompt message is used to indicate that the target control command has been started.

[0103] When this tapping function is used for the first time after it has been enabled, a fourth prompt message is sent to the smart device. This fourth prompt message indicates that the target control command has been started.

[0104] For example, the fourth prompt message could be to execute the target control command and announce, "The target control program has started. You can change the startup program in the smart device's smart assistant!"

[0105] S310: Send a third prompt message to the smart device. The third prompt message is used to prompt one or more of the following: the target control command has been started, and the method of changing the control command corresponding to the tap function.

[0106] When this use of the tap function is not the first time it has been used since the tap function was enabled, a third prompt message is sent to the smart device. The third prompt message is used to indicate one or more of the following: the target control command has been executed, and the change method of the control command corresponding to the tap function.

[0107] Once it is confirmed that the tapping function has not been used for the first time, the cloud server will generate a third prompt message. This message should explicitly include confirmation that "the target control command has been executed," and, if necessary, include explanations or instructions on "how to change the control command corresponding to the tapping function." Subsequently, the cloud server will send this prompt message to the smart device or its associated user interface via appropriate communication protocols and channels (such as Bluetooth, Wi-Fi, etc.).

[0108] To enhance user experience, cloud servers can allow users to modify the control commands associated with keystrokes. For example, a user-friendly interface or menu option can be provided, allowing users to customize the mapping between keystroke actions and control commands. Additionally, a brief explanation of how to change the settings or a link to the relevant settings page can be provided directly in the third-party prompt message.

[0109] For example, the third prompt message could be to execute the target control command and announce "Target control program has started!".

[0110] S311: If the smart device is in working state, a fifth prompt message is sent to the smart device. The fifth prompt message is used to prompt the user to control the smart device through the tap function when the smart device is in standby state.

[0111] The operating status is used to indicate that the smart device is already performing its functions.

[0112] When the tap function of the smart device is enabled and the smart device is in working condition, a fifth prompt message is sent to the smart device. The fifth prompt message is used to prompt the user to control the smart device through the tap function when it is in standby mode.

[0113] First, by informing users in advance about the validity of the tapping function in standby mode, users can more flexibly plan and control the use of smart devices. Second, this prompt helps reduce user confusion and accidental operation, because not all users are aware that tapping is ineffective when the device is in operation, or are unclear about the specific function of tapping in standby mode. Finally, by proactively providing useful information, it helps users better understand and use the various functions of smart devices.

[0114] For example, the fifth prompt message could be: "The smart device is working. The message could not be sent. To change the program, try the smart device's smart assistant!"

[0115] The multimodal interaction method for smart devices provided in this embodiment receives tapping information sent by the smart device. This tapping information is obtained by the voice module in the smart device from the user's tapping of the smart device. The tapping information is analyzed to determine a first confidence level, which represents the degree to which the tapping information can trigger control commands. If the first confidence level is greater than a preset threshold, it is determined whether the tapping function of the smart device is enabled. If not, the smart device is in standby mode, and a first prompt message is sent to the smart device to prompt the user whether to enable the tapping function. The method also receives and analyzes interactive voice from the smart device. If the interactive voice indicates that the tapping function is enabled, the tapping function of the smart device is enabled, and a second prompt message is sent to the smart device. The second prompt message indicates one or more of the following: the tapping function is enabled, the tapping gesture, and the control command corresponding to the tapping function. The method of changing the command; if so, the smart device is in standby mode, then the target control command corresponding to the tap function is queried and sent to the smart device. If this is the first use of the tap function after it has been enabled, a third prompt message is sent to the smart device. The third prompt message is used to indicate one or more of the following: the target control command has started to be executed, and the method of changing the control command corresponding to the tap function. If this is not the first use of the tap function after it has been enabled, a fourth prompt message is sent to the smart device. The fourth prompt message is used to indicate that the target control command has started to be executed. If the tap function of the smart device is enabled, the smart device is in working mode, and a fifth prompt message is sent to the smart device. The fifth prompt message is used to remind the user to control the smart device through the tap function when it is in standby mode. This method improves the convenience of controlling the smart device while ensuring the accuracy of operation, and realizes a highly intelligent operating experience.

[0116] Figure 4 This is the flow chart of the multimodal interaction method for smart devices provided in this application. Figure 3 The executing entity in this embodiment is, for example, an intelligent device control system. Figure 4 As shown, the multimodal interaction method for smart devices provided in this embodiment includes:

[0117] S401: In response to a tapping message triggered by a user tapping a smart device, identify a second confidence level of the tapping message, the second confidence level representing the degree to which the tapping message is used to trigger a control command.

[0118] For example, the second confidence level can be 0.9 or 0.92. This application does not specify a particular limit on the second confidence level.

[0119] In practical applications, smart devices may receive various unintended tap or vibration signals, such as ambient noise or accidental touches. If these non-target signals are misinterpreted as control commands, it can lead to malfunctions of the smart device, affecting user experience and even creating safety hazards. Therefore, by introducing a second confidence level, the smart device control system can more accurately determine which tap information truly represents the user's control intent, thereby effectively filtering out invalid or accidental touches and improving the intelligence and stability of the smart device.

[0120] Understandably, smart devices need to incorporate highly sensitive sensors (such as accelerometers or vibration sensors) to capture the user's tapping motions and convert them into electrical signals. Secondly, the smart device control system needs to preprocess the received signals, including noise reduction and filtering, to improve signal quality. Next, complex algorithms (such as machine learning models) are used to extract features and perform pattern recognition on the processed signals to distinguish different types of tapping motions. Based on this, a second confidence level is calculated. Finally, based on the level of the second confidence level, the smart device control system decides whether to interpret the current tapping information as a valid control command and execute the corresponding operation.

[0121] S402: If the second confidence level is greater than the preset threshold, send the tapping information to the cloud server.

[0122] First, by setting preset thresholds, false alarms or low-confidence information can be effectively filtered out, ensuring that the data sent to the cloud server has high accuracy and reliability, thereby reducing unnecessary data processing burdens and storage costs. Second, timely uploading of confirmed information to the cloud server enables centralized data management and remote access, facilitating monitoring and response by administrators at any time. Furthermore, the powerful data processing capabilities of the cloud server support more complex analyses, such as pattern recognition and anomaly detection, providing strong support for further optimizing the performance of intelligent device control systems.

[0123] S403: Receives target control commands sent by the cloud server. These target control commands are used to control the operation of smart devices.

[0124] When the tap function of a smart device is enabled and the device is in standby mode, the smart device control system receives information from the cloud server. This information is intended to prompt the user to consider how to control the operation of the smart device.

[0125] Optionally, after this step, the system receives the user's interactive voice and sends the interactive voice to the cloud server; it also receives a second prompt message from the cloud server, which is used to prompt one or more of the following: the tapping function has been enabled, the tapping gesture, and the method of changing the control command corresponding to the tapping function.

[0126] The smart device first receives interactive voice messages from the user, which may contain instructions for operating the device's functions or query requests. The smart device then sends these voice messages to a cloud server for processing. Upon receiving the voice messages, the cloud server recognizes and analyzes them, and generates corresponding secondary prompts based on the user's intent. These secondary prompts may include confirmation that the device's tap function is enabled, instructions on the gestures required for the tap function, or guidance on how to change the control commands corresponding to the tap function.

[0127] Optionally, this step also includes: after the smart device is powered on, if the tapping function is not enabled, outputting a sixth prompt message, which is used to prompt how to enable the tapping function; if the tapping function is enabled but not used within a preset time, outputting a seventh prompt message, which is used to prompt the gesture for the tapping function.

[0128] After a smart device is powered on, it automatically outputs different prompts to the user based on the activation and usage status of its tapping function. Specifically, if the smart device detects that the tapping function is not activated after powering on, it will output a sixth prompt, which aims to guide the user to understand and activate the tapping function. On the other hand, if the tapping function is activated but has not been used within a preset time period, the device will output a seventh prompt, which reminds the user of the tapping gesture so that the user can familiarize themselves with or utilize the function again.

[0129] For example, the sixth prompt could be "The tapping function is now available! After enabling the function, tap the top drum door to start washing clothes directly"; the seventh prompt could be "Tap the top drum door twice to start washing clothes, come and try it!"

[0130] The multimodal interaction method for smart devices provided in this embodiment responds to tapping information triggered by a user tapping the smart device, identifies a second confidence level of the tapping information, and the second confidence level characterizes the degree to which the tapping information can trigger control commands. When the second confidence level is greater than a preset threshold, the tapping information is sent to a cloud server, and the target control command sent by the cloud server is received. The target control command is used to control the operation of the smart device. This method achieves more efficient and accurate control of smart devices through the collaborative work of the cloud server and the smart device, bringing users a more convenient and comfortable life experience.

[0131] Figure 5 This is a structural schematic diagram of the multimodal interaction device for the intelligent device provided in this application. For example... Figure 5 As shown, this application provides a multimodal interaction device for a smart device. The multimodal interaction device 500 for the smart device includes:

[0132] The receiving module 501 is used to receive tapping information sent by the smart device. The tapping information is obtained by the voice module in the smart device from the interactive information generated by the user tapping the smart device.

[0133] The determination module 502 is used to perform sound analysis on the tapping information and determine the first confidence level of the tapping information. The first confidence level represents the degree to which the tapping information can trigger control commands.

[0134] The judgment module 503 is used to determine whether the tapping function of the smart device is enabled when the first confidence level is greater than a preset threshold.

[0135] The sending module 504 is used to query the target control command corresponding to the tapping function and send the target control command to the smart device if the tapping function of the smart device is enabled and the smart device is in standby mode.

[0136] Optionally, the sending module 504 is further configured to send a first prompt message to the smart device if the tapping function of the smart device is not enabled and the smart device is in standby mode. The first prompt message is used to prompt the user whether to enable the tapping function.

[0137] Optionally, the receiving module 501 is also used to receive interactive voice sent by the smart device and parse the interactive voice;

[0138] The sending module 504 is further configured to enable the tapping function of the smart device if the interactive voice instruction enables the tapping function, and send a second prompt message to the smart device. The second prompt message is used to prompt one or more of the following: the tapping function has been enabled, the gesture of the tapping function, and the change method of the control command corresponding to the tapping function.

[0139] Optionally, the judgment module 503 is also used to determine whether the current use of the tapping function is the first use after the tapping function has been enabled;

[0140] The sending module 504 is also used to send a third prompt message to the smart device if it is the first time it is used. The third prompt message is used to prompt one or more of the following: the target control command has been started, and the change method of the control command corresponding to the tap function.

[0141] The sending module 504 is also used to send a fourth prompt message to the smart device if it is not the first time it is used. The fourth prompt message is used to indicate that the target control command has been started.

[0142] Optionally, the sending module 504 is also used to send a fifth prompt message to the smart device if the tapping function of the smart device is enabled and the smart device is in working state. The fifth prompt message is used to prompt the user to control the smart device through the tapping function when the smart device is in standby state.

[0143] Figure 6 This is a structural schematic diagram of the multimodal interaction device of the intelligent device provided in this application. For example... Figure 6 As shown, this application provides a multimodal interaction device for a smart device. The multimodal interaction 600 of the smart device includes: a receiver 601, a transmitter 602, a processor 603, and a memory 604.

[0144] Receiver 601 is used to receive instructions and data;

[0145] Transmitter 602 is used to send commands and data;

[0146] Memory 604 is used to store instructions executed by the computer;

[0147] The processor 603 is used to execute computer execution instructions stored in the memory 604 to implement the various steps performed by the multimodal interaction method of the smart device in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the multimodal interaction method of the smart device.

[0148] Alternatively, the memory 604 can be either standalone or integrated with the processor 603.

[0149] When the memory 604 is set up independently, the electronic device also includes a bus for connecting the memory 604 and the processor 603.

[0150] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the multimodal interaction method of the smart device as described above.

[0151] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A multi-modal interaction method of a smart device, the method comprising: The method comprises the following steps: receiving knocking information sent by the smart device, wherein the knocking information is obtained by a voice module in the smart device and represents interaction information generated by a user knocking the smart device; performing sound analysis on the knocking information to determine a first confidence degree of the knocking information, wherein the first confidence degree represents a degree to which the knocking information is used to trigger a control instruction; if the first confidence degree is greater than a preset threshold, determining whether a knocking function of the smart device is enabled, and if the knocking function of the smart device is enabled and the smart device is in a standby state, querying a target control instruction corresponding to the knocking function and sending the target control instruction to the smart device.

2. The method of claim 1, wherein, The method further comprises the following steps: if the knocking function of the smart device is not enabled and the smart device is in the standby state, sending first prompt information to the smart device, wherein the first prompt information is used to prompt the user whether to enable the knocking function.

3. The method of claim 2, wherein, The method further comprises the following steps: receiving an interaction voice sent by the smart device and performing analysis on the interaction voice; if the interaction voice indicates that the knocking function is enabled, enabling the knocking function of the smart device and sending second prompt information to the smart device, wherein the second prompt information is used to prompt one or more of the following information: the knocking function has been enabled, a gesture of the knocking function, and a change mode of a control instruction corresponding to the knocking function.

4. The method of claim 3, wherein, The method further comprises the following steps: determining whether this use of the knocking function is the first use after the knocking function is enabled; if it is the first use, sending third prompt information to the smart device, wherein the third prompt information is used to prompt one or more of the following information: the target control instruction has started to be executed, and the change mode of the control instruction corresponding to the knocking function; if it is not the first use, sending fourth prompt information to the smart device, wherein the fourth prompt information is used to prompt that the target control instruction has started to be executed.

5. The method of claim 1, wherein, The method further comprises the following steps: if the knocking function of the smart device is enabled and the smart device is in a working state, sending fifth prompt information to the smart device, wherein the fifth prompt information is used to prompt the user to control the smart device through the knocking function when the smart device is in the standby state. 6.A method for multi-modal interaction of a smart device, the method comprising: The method comprises the following steps: in response to knocking information triggered by a user knocking the smart device, identifying a second confidence degree of the knocking information, wherein the second confidence degree represents a degree to which the knocking information is used to trigger a control instruction; if the second confidence degree is greater than a preset threshold, sending the knocking information to a cloud server; receiving a target control instruction sent by the cloud server, wherein the target control instruction is used to control the operation of the smart device.

7. The method of claim 6, wherein, The method further comprises the following steps: receiving an interaction voice of the user and sending the interaction voice to the cloud server; receiving second prompt information sent by the cloud server, wherein the second prompt information is used to prompt one or more of the following information: the knocking function has been enabled, a gesture of the knocking function, and a change mode of a control instruction corresponding to the knocking function.

8. The method of claim 6, wherein, The method further comprises the following steps: after the smart device is powered on, if the knocking function is not enabled, outputting sixth prompt information, wherein the sixth prompt information is used to prompt a method of enabling the knocking function. If the knocking function is enabled, but the knocking function is not used within the preset time, seventh prompt information is output, the seventh prompt information being used for prompting a gesture of the knocking function.

9. A computer readable storage medium, characterized in that, The computer readable storage medium comprises a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 8. 10.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 8 by using the computer program.

Images