Voice remote control method and remote control system
By combining scene information and multimodal scene understanding model, the accurate understanding and response of user commands is achieved, and dynamic learning and optimization is used by user feedback, the problem of lack of feedback and adaptability of voice remote control systems in the existing technology is solved, and the problem of improving the The accuracy and user experience of voice remote control.
Patent Information
- Application Number
- CN202410943845.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing voice remote control system lacks effective feedback and adaptive learning mechanisms, which makes it difficult for users to accurately control smart devices, and cannot adjust them in time when users are not satisfied with their feedback.
By combining scene information and multimodal scene understanding model, the accurate understanding and response of user commands is achieved, and dynamic learning and optimization is used to use user feedback, and the device control command mapping table is updated to improve the accuracy and user experience of voice remote control.
It improves the accuracy and user experience of voice remote control, realizes adaptive learning and optimization of voice remote control system, meets users' personalized preferences, and provides a more intuitive and smooth control experience.
Smart Images

Figure CN118942455A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of smart home control, and in particular to a voice remote control method and a remote control system. Background Art
[0002] With the rapid development of smart home and IoT technologies, users are increasingly demanding the ability to perform various operations via voice remote control devices. As a natural way of interaction, voice remote control provides users with a convenient life experience, especially when their hands are occupied or their vision is not focused.
[0003] In the related technical solution, the voice remote control system converts the user's voice command into specific device control instructions, which are then sent to the corresponding smart device to perform the operation requested by the user. Such a technical solution enables users to control smart devices at home through simple voice commands, such as turning on and off lights, adjusting the temperature, etc.
[0004] Although the relevant technology provides basic voice control functions, there are still certain limitations in practical applications. After receiving voice commands, the existing system often lacks effective feedback and adaptive learning mechanisms, resulting in users being unable to know whether the device responds correctly in time, and unable to enable the voice remote control system to continuously learn and update, making it difficult for users to accurately control the device by voice. Summary of the invention
[0005] The present application provides a voice remote control method and a remote control system, which accurately respond to user commands by combining scene information, a multimodal scene understanding model, and a command mapping relationship, while realizing dynamic learning and adaptive adjustment of the voice remote control system, thereby improving the accuracy of the voice remote control.
[0006] In a first aspect, the present application provides a voice remote control method, which is applied to a remote control system, and the method includes: performing voice recognition on a received first user voice to obtain a user command converted from voice form to text form; obtaining current scene information; inputting the user command and the current scene information into a preset multimodal scene understanding model to obtain user needs, and the user needs include user intentions, operation objects, and operation parameters; based on a preset user demand control instruction mapping table, obtaining a device control instruction corresponding to the user command according to the user needs, and the user demand control instruction mapping table includes the correspondence between the user needs and the control instruction; sending the device control instruction to the corresponding controlled device to control the controlled device to perform the operation; sending the received execution result of the controlled device to the user and obtaining user feedback on the execution result; if the user feedback is dissatisfied, prompting the user to input a second user voice; if the user feedback is satisfied, the current scene information, the user command, and the device control instruction are associated and recorded to update the multimodal scene understanding model and the user demand control instruction mapping table.
[0007] In the above embodiment, the remote control system combines user commands, current scene information, a multimodal scene understanding model, and a user demand control instruction mapping table to achieve accurate understanding and response to the first user's voice command. After determining the user's needs, the remote control system sends the corresponding device control instructions to the controlled device to control the controlled device to perform operations. After that, the remote control system obtains user feedback. If the user feedback is not satisfactory, it means that the effect of the voice remote control is not what the user wants, then the remote control system will re-acquire the second user's voice and re-perform voice remote control; if the user feedback is satisfactory, it means that the effect of the voice remote control is what the user wants, then the remote control system dynamically adjusts the multimodal scene understanding model and the user demand control instruction mapping table through this voice remote control, thereby achieving adaptive learning and optimization of the voice remote control system. Compared with the prior art, this method makes full use of scene information and user feedback to improve the accuracy of voice remote control and user experience.
[0008] In combination with some embodiments of the first aspect, in some embodiments, user commands and current scene information are input into a preset multimodal scene understanding model to obtain user needs, which include user intentions, operation objects, and operation parameters. Specifically, the following steps are performed: converting user commands and current scene information into user command vectors and current scene vectors; connecting the user command vector with the current scene vector to obtain a command-scene joint vector; inputting the command-scene joint vector into the multimodal scene understanding model to obtain a structured representation of user needs as user needs, and the user needs representation includes user intentions, operation objects, and operation parameters.
[0009] In the above embodiment, the implementation method of the multimodal scene understanding model is refined. This multimodal fusion method based on vector representation can better mine the semantic association between user commands and current scene information, thereby achieving accurate understanding of the user's true intentions. Compared with the understanding method based solely on voice command text, this method introduces scene information as an important context clue, greatly eliminating the ambiguity of voice commands and improving the accuracy of semantic understanding.
[0010] In combination with some embodiments of the first aspect, in some embodiments, based on a preset user demand control instruction mapping table, device control instructions corresponding to user commands are obtained according to user needs, and the user demand control instruction mapping table includes the correspondence between user needs and control instructions, specifically including: inputting user intentions, operation objects, and operation parameters into a preset instruction generation network; based on the user demand control instruction mapping table, generating device control instructions corresponding to user commands through the instruction generation network.
[0011] In the above embodiment, the implementation method of device control instruction generation is refined. This end-to-end instruction generation method avoids the cumbersome matching process based on rules or templates, and has better flexibility and generalization ability. At the same time, due to the use of an instruction generation network, learning and optimization can be carried out in a data-driven manner to continuously improve the accuracy and adaptability of device control instructions. Compared with the traditional instruction matching method, this technical solution realizes the direct mapping of user needs to control instructions, shortens the response path, and improves the real-time and efficiency of voice remote control.
[0012] In combination with some embodiments of the first aspect, in some embodiments, based on a preset user demand control instruction mapping table, device control instructions corresponding to user commands are obtained according to user needs, and the user demand control instruction mapping table includes the correspondence between user needs and control instructions, specifically including: determining candidate devices related to current scene information in a preset device association map, the device association map includes each device node and its functional attributes, and the association relationship between device nodes; matching user intentions and operation objects with candidate devices and their corresponding functional attributes to determine controlled devices; and matching device control instructions corresponding to user commands from the user demand control instruction mapping table according to the controlled device and operating parameters.
[0013] In the above embodiment, the device association map is introduced in the process of generating device control instructions, which further improves the accuracy of instruction matching. The device association map fully considers the correlation between devices and the operation context, and can determine the controlled device by matching the user's intention and the operation object with the candidate device and its corresponding functional attributes, thereby more intelligently inferring the user's real control intention and improving the accuracy and robustness of voice remote control.
[0014] In combination with some embodiments of the first aspect, in some embodiments, before performing voice recognition on the received first user voice to obtain a user command converted into text form, the method also includes: obtaining a historical status record of the device, the historical status record including the parameter setting status of the device under different scenario information; determining a common mode of the device according to the historical status record as a user's personalized preference.
[0015] In the above embodiment, the remote control system introduces user personalized preferences in the voice remote control process, providing a more humane and intelligent interactive experience. Through the historical status record of the device, the remote control system can determine the parameter setting status of the device under different scene information, thereby determining the common mode of the device, and obtaining the user's personalized preferences. It can accurately characterize the user's inclination towards device functions and parameters to achieve customized control of the device and bring a smoother operating experience. Compared with traditional voice remote control solutions, this technical solution integrates user portraits and personalized recommendation technologies, allowing the remote control system to provide customized intelligent services based on the user's unique habits and preferences, thereby improving the user experience.
[0016] In combination with some embodiments of the first aspect, in some embodiments, after the step of inputting user commands and current scene information into a preset multimodal scene understanding model to obtain user needs, wherein the user needs include user intentions, operation objects, and operation parameters, the method further includes: calculating the degree of match between user needs and user personalized preferences; when the degree of match exceeds a preset degree of match threshold, asking the user whether to set the device to a common mode; and after receiving confirmation feedback from the user, executing the operation of setting the device to a common mode.
[0017] In the above embodiments, the application mode of user personalized preferences is refined, providing a more intelligent and active human-computer interaction experience. After generating user needs, the remote control system will calculate the matching degree between user needs and user preferences. When the matching degree exceeds the preset matching degree threshold, the remote control system will actively initiate a confirmation inquiry to the user, prompting the user whether the device needs to be set to the common mode. This intelligent inquiry mechanism is a kind of prejudgment and completion of user intentions, helping users to omit the steps of explicit commands. Users only need to make simple positive feedback to the prompts of the remote control system, and they can open the personalized scene with one click, quickly configure the device to the most familiar and favorite state, so as to obtain a more intuitive and smooth control experience. This human-computer interaction mode breaks through the passive response limitations of traditional voice control. The system actively asks questions to the user and guides the user to confirm feedback, thereby realizing a hybrid active interaction form. The remote control system intervenes in the user's control process in a timely manner, and provides accurate control suggestions based on the user's previous behavior habits, which reduces the user's cognitive and operational burden while also increasing the user's sense of use.
[0018] In combination with some embodiments of the first aspect, in some embodiments, after the step of sending the received execution result of the controlled device to the user and obtaining user feedback on the execution result, the method also includes: recording user feedback and corresponding device control instructions to build a user satisfaction database; regularly identifying the relationship between device control instructions and user feedback to obtain analysis results; and adjusting the user satisfaction database based on the analysis results.
[0019] In the above embodiment, the remote control system builds a user satisfaction database, and then optimizes and adjusts the multimodal scene understanding model and the user demand control command mapping table based on it, strengthens the command combination of positive feedback, and weakens or corrects the combination of negative feedback. Through this feedback-driven self-optimization mechanism, the voice remote control system can continuously accumulate experience, learn and evolve autonomously during operation, and dynamically adapt to the changing needs of users. Compared with the static voice solution, this solution embeds closed-loop feedback, allowing the remote control system to continuously learn and grow, thereby strengthening the adaptive learning mechanism.
[0020] In a second aspect, an embodiment of the present application provides a remote control system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the remote control system to execute the method described in the first aspect and any possible implementation method of the first aspect.
[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a remote control system, enables the remote control system to execute the method described in the first aspect and any possible implementation of the first aspect.
[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a remote control system, enables the remote control system to execute the method described in the first aspect and any possible implementation of the first aspect.
[0023] It is understandable that the remote control system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.
[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The remote control system combines user commands, current scene information, multimodal scene understanding models, and user demand control command mapping tables to achieve accurate understanding and response to the first user's voice commands. After determining the user's needs, the remote control system sends the corresponding device control instructions to the controlled device to control the controlled device to perform operations. After that, the remote control system obtains user feedback. If the user feedback is not satisfactory, it means that the effect of the voice remote control is not what the user wants, then the remote control system will re-acquire the second user's voice and re-perform voice remote control; if the user feedback is satisfactory, it means that the effect of the voice remote control is what the user wants, then the remote control system dynamically adjusts the multimodal scene understanding model and the user demand control command mapping table through this voice remote control, thereby achieving adaptive learning and optimization of the voice remote control system. Compared with the prior art, this method makes full use of scene information and user feedback to improve the accuracy of voice remote control and user experience.
[0025] 2. The remote control system will calculate the matching degree between user needs and user preferences. When the matching degree exceeds the preset matching degree threshold, the remote control system will actively initiate a confirmation inquiry to the user, prompting the user whether the device needs to be set to the common mode. This intelligent inquiry mechanism is a kind of prediction and completion of user intentions, helping users to omit the steps of explicit commands. Users only need to make simple positive feedback to the prompts of the remote control system, and they can open personalized scenes with one click, quickly configure the device to the most familiar and favorite state, and obtain a more intuitive and smooth control experience. This human-computer interaction mode breaks through the passive response limitations of traditional voice control. The system actively asks questions to the user and guides the user to confirm feedback, thus realizing a hybrid active interaction form. The remote control system intervenes in the user's control process in a timely manner, and provides accurate control suggestions based on the user's previous behavior habits, which reduces the user's cognitive and operational burden while also increasing the user's sense of use. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is a flow chart of the voice remote control method in the embodiment of the present application; Figure 2 This is another flowchart of the voice remote control method in the embodiment of the present application; Figure 3 It is a schematic diagram of the structure of a physical device of the remote control system in an embodiment of the present application. DETAILED DESCRIPTION
[0027] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.
[0028] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.
[0029] See also Figure 1 , which is a flow chart of the voice remote control method in an embodiment of the present application.
[0030] S101, performing voice recognition on a received first user voice to obtain a user command converted from a voice form to a text form; After the remote control system receives the first user's voice through the voice receiving module, it first needs to process and recognize the first user's voice. The remote control system can use various voice recognition technologies, such as statistical methods based on acoustic models and language models, to convert the first user's voice from voice form to user commands in text form. In the voice recognition process, the remote control system needs to combine multiple technologies such as environmental noise elimination, voiceprint recognition, sentence segmentation, error correction, etc. to improve the accuracy and robustness of recognition. For example, the remote control system can use a microphone array to locate the user's sound source position and adaptively suppress the environmental noise to obtain a purer first user voice. In addition, the remote control system can also verify the identity of the speaker through voiceprint recognition technology to prevent malicious control by illegal users.
[0031] S102, obtaining current scene information; In order to understand the user's intention more accurately, the remote control system needs to further obtain the current scene information. The current scene information here can include various contextual factors related to user interaction, such as the user's location, time, surrounding environment, device status, etc. The remote control system can collect current scene information through a variety of sensors and data interfaces. For example, through GPS or indoor positioning system, the user's current geographical location and spatial area can be obtained; through the status report of smart home devices, the working mode and parameter configuration of different devices can be understood; by analyzing the user's schedule and behavioral habits, the user's current activity status and context can be inferred. In addition, the remote control system can also actively conduct multiple rounds of dialogue with the user and obtain more scene details by asking questions, such as "Which room are you in now?" "Which channel do you want to watch?" etc. The more comprehensive and detailed the current scene information obtained by the remote control system, the more accurately it can grasp the true intention of the user's command and provide a more reasonable and appropriate control strategy. It should be emphasized that the acquisition of current scene information needs to be carried out under the premise of respecting user privacy. The remote control system should provide users with necessary privacy control options so that users can independently manage the disclosure scope of personal data.
[0032] S103, inputting the user command and the current scene information into a preset multimodal scene understanding model to obtain the user demand, which includes the user intention, the operation object and the operation parameter; When the remote control system obtains the user command and the current scene information, it needs to perform semantic understanding and intent analysis to accurately grasp the user's needs, which include user intent, operation object, and operation parameters. For example, the user says to a smart home remote control system: "Dim the lights in the living room to 30%." In this case, the user command is "Dim the lights in the living room to 30%", and the current scene information is obtained through environmental sensors, such as the current light brightness, time, whether someone is in the room, etc. Application of multimodal scene understanding model: First, use NLP tools to perform word segmentation and semantic analysis on user commands, and convert user commands into numerical vectors through word vector models (such as Word2Vec or BERT) to capture semantic information. Then, use sensor data to obtain current scene information, such as understanding the current brightness level through light sensors. If there is a camera, image recognition technology can be used to determine whether there are people in the room and their activity status. The multimodal scene understanding model combines information from text and sensors, and applies an attention mechanism to enable the multimodal scene understanding model to focus on the most important information for the current task (for example, focus on keywords such as "dim" and "30%"). Based on the user intention analyzed by the multimodal scene understanding model, the intelligent system adjusts the brightness of the living room lights to 30%.
[0033] User demand analysis: User intent: Adjust light brightness Object of operation: Living room lamp Operating parameters: Brightness adjusted to 30%.
[0034] In the embodiment of the present application, a preset multimodal scene understanding model is used to comprehensively analyze information of two different modes, namely user commands and current scene information. The multimodal scene understanding model can use various natural language processing and machine learning techniques, such as word vectors, attention mechanisms, memory networks, etc., to establish semantic associations between user commands and current scene information.
[0035] Method for constructing a multimodal scene understanding model: Collect a large amount of scene-related text descriptions, scene-related video data, and remote control commands issued by users. Perform text preprocessing and visual preprocessing on the above data. Text preprocessing includes word segmentation, removal of stop words, part-of-speech tagging, etc. Visual preprocessing includes adjusting image size and normalization, etc. Extract text features from text data through the BoW model, and use 3D CNN or LSTM to extract image features from video data. Merge the features of all modalities at the input level, merge the results of each modality at the decision level, realize feature fusion at the middle layer, and use attention mechanism or other technologies to strengthen key information. Select a suitable machine learning or deep learning framework to train the preset model. When the loss function of the preset model exceeds the preset loss function threshold, it is confirmed that the multimodal scene understanding model has been obtained.
[0036] Optionally, in general, user commands and current scene information are input into a preset multimodal scene understanding model, and user needs can be obtained in the following way: converting user commands and current scene information into user command vectors and current scene vectors; connecting the user command vector with the current scene vector to obtain a command-scene joint vector; inputting the command-scene joint vector into the multimodal scene understanding model to obtain a structured representation of user needs as user needs, and the user need representation includes user intentions, operation objects, and operation parameters.
[0037] Use natural language processing technology (such as word embedding, BERT, etc.) to convert user commands into numerical vectors, namely user command vectors. Convert the acquired scene information into vector form, namely current scene vector, through a specific image processing model (such as CNN) or other perception models.
[0038] The user command vector and the current scene vector are combined into a single vector by some means (such as splicing), namely the command-scene joint vector. The command-scene vector should contain sufficient information to represent the state of the entire command scene.
[0039] The multimodal scene understanding model receives the command-scene joint vector, analyzes and processes it, and identifies and extracts user intent, operation objects, and operation parameters from the command joint vector.
[0040] The output of the multimodal scene understanding model is usually the classification of user intentions, the identification of operation objects and the specific values of operation parameters. These outputs together constitute a complete description of user needs, that is, a structured representation of user needs, as user needs.
[0041] For example, in a smart home environment, a user might say, "Turn down the air conditioner in the living room." The user command is "Turn down the temperature," and the scene information includes the current indoor temperature, air conditioner status, etc. The user command vector and the scene vector are combined and input into the model, which needs to identify the operation intention (turn down the temperature), the operation object (the air conditioner in the living room), and the operation parameters (the new temperature value or the adjustment amount).
[0042] S104, based on a preset user demand control instruction mapping table, obtaining a device control instruction corresponding to the user command according to the user demand, wherein the user demand control instruction mapping table includes a correspondence between the user demand and the control instruction; After obtaining user needs, the remote control system needs to further convert these user needs into specific device control instructions. The remote control system pre-builds a user demand control instruction mapping table, which defines the correspondence between different user needs and control instructions. For example, when the user demand is "turn on the bedroom light", the corresponding device control instruction may be "bedroom lamp / switch / turn on"; and when the user demand is "raise the living room temperature by 3 degrees", the corresponding device control instruction may be "living room air conditioner / temperature / +3". The remote control system automatically searches for control instruction entries that match the current user needs based on this user demand control instruction mapping table. This matching process can use simple keyword matching or more complex semantic matching algorithms, such as reasoning based on knowledge graphs, semantic similarity calculation based on deep learning, etc. In the matching process, the remote control system must not only consider the functional semantics of the control instructions, but also consider technical details such as the brand model, interface protocol, etc. of different devices to ensure that the generated control instructions can be correctly parsed and executed. It should be noted that as the types and functions of smart home devices continue to increase, and users express themselves in a variety of ways, the user demand control command mapping table needs to be continuously expanded and optimized to adapt to different application scenarios. In addition, for some complex user needs, a single control command may not be able to fully meet them, and a command sequence or workflow needs to be generated to achieve them. This requires that the user demand control command mapping table should not only support one-to-one static mapping, but also support one-to-many and many-to-many dynamic combinations to cope with more flexible and changeable control needs.
[0043] Optionally, under normal circumstances, based on a preset user demand control instruction mapping table, obtaining device control instructions corresponding to user commands according to user needs can be achieved in the following way: inputting user intentions, operation objects, and operation parameters into a preset instruction generation network; based on the user demand control instruction mapping table, generating device control instructions corresponding to user commands through the instruction generation network.
[0044] Optionally, under normal circumstances, based on a preset user demand control instruction mapping table, obtaining device control instructions corresponding to user commands according to user needs can be achieved in the following way: determining candidate devices related to current scene information in a preset device association map, the device association map including each device node and its functional attributes, and the association relationship between device nodes; matching user intentions and operation objects with candidate devices and their corresponding functional attributes to determine the controlled device; matching device control instructions corresponding to user commands from the user demand control instruction mapping table according to the controlled device and operating parameters.
[0045] First, the remote control system needs to build a device association graph that contains all device nodes, their functional attributes, and the relationship between nodes. The device association graph can be used to represent the connectivity between different devices and their functions in a specific environment. The construction of the device association graph usually includes the following: Device nodes: Each node represents a device, such as lamps, air conditioners, curtains, etc. Functional attributes: the attributes of each device node, such as temperature adjustment, switch status, brightness adjustment, etc. Association: The relationship between devices, such as the relationship between a light fixture and a specific room, or the relationship between an air conditioner and a temperature sensor.
[0046] After obtaining the structured representation of user needs (including user intent, operation object, and operation parameters), the remote control system selects candidate devices from the device association map based on this information: User intent analysis: Determine the user's specific intent, such as "turn down", "turn on", etc.
[0047] Operation object matching: Find the device node that matches the operation object specified by the user (such as "living room air conditioner") in the device association map.
[0048] Determine the controlled device: Combine the user's operation intention and the device nodes in the device association map to determine the final controlled device. This step may involve: screening all possible devices from the device association map and ensuring that the functional attributes of the device can respond to the user's operation intention and parameters (such as air conditioning equipment that can adjust the temperature).
[0049] According to the determined controlled device and operating parameters, the corresponding device control instructions are matched from the preset user demand control instruction mapping table. This user demand control instruction mapping table should contain the following information: Device type: The control instruction set corresponding to each device.
[0050] Operation type: The control instructions corresponding to each operation (such as on / off, adjustment, etc.).
[0051] Parameter mapping: control instructions for specific operating parameters such as temperature value, brightness, etc.
[0052] Finally, the remote control system sends the generated device control instructions to the corresponding controlled device to execute the user's command. This usually involves IoT communication technologies such as Wi-Fi, Zigbee, Bluetooth, etc.
[0053] S105, sending the device control instruction to the corresponding controlled device to control the controlled device to perform an operation; After generating device control instructions corresponding to user needs, the remote control system needs to send these device control instructions to the corresponding controlled devices to achieve remote control of the controlled devices. The key here is to establish a reliable and efficient communication mechanism to ensure that the device control instructions can be accurately transmitted to the controlled devices. The remote control system usually needs to support multiple communication protocols and standards, such as WiFi, Bluetooth, ZigBee, infrared, etc., to adapt to various types of smart hardware on the market. At the same time, the remote control system also needs to implement necessary security mechanisms, such as encryption authentication, authority management, etc., to prevent instructions from being maliciously hijacked or tampered with. In the actual instruction sending process, the remote control system needs to dynamically adjust the sending strategy, such as retransmission, timeout, cache, etc., according to factors such as the online status of the controlled device and network quality, in order to improve the success rate and real-time performance of the control. For example, for some devices with slow response, the remote control system can introduce an asynchronous instruction mode to cache the device control instructions first and execute them in batches when the device is idle; for some important or high-priority device control instructions, the remote control system can adopt a special guaranteed delivery mechanism to improve the arrival rate of instructions through retransmission or multi-path.
[0054] S106, sending the received execution result of the controlled device to the user and obtaining user feedback on the execution result; The remote control system needs to track the execution status of the controlled device and promptly feedback the execution results to the user to ensure that the user's needs are properly met, while also allowing the user to understand the working status of the controlled device in real time. Specifically, after the controlled device completes the specified operation, it needs to actively report the execution results to the remote control system. The execution results can include the success / failure status of the operation, the current parameter values of the controlled device, and the real-time monitoring information of the environment. After receiving this data, the remote control system needs to present it in a user-friendly way, such as voice broadcast, graphical interface update, push notification, etc., so that the user can intuitively understand the control effect. At the same time, the remote control system should also provide users with channels for feedback and evaluation, and collect users' subjective opinions such as satisfaction ratings and correction suggestions for the execution results. These user feedbacks are crucial to improving the performance and user experience of the remote control system.
[0055] S107, if the user feedback is dissatisfied, prompting the user to input a second user voice; If the user feedback is not satisfied, it means that the control effect is not good. The remote control system needs to prompt the user to re-enter the second user's voice to further clarify or correct the control requirements. Unlike traditional graphical interfaces, voice interaction allows users to supplement or correct previous voice commands at any time, thereby achieving a more natural and smooth human-computer dialogue. For example, when a user is dissatisfied with the volume adjustment result of the TV, he can directly say "turn it up a little more" or "it's a bit too loud" without repeating the previous complete command. When guiding the user to input the second user's voice, the remote control system can adopt some friendly interactive designs, such as voice prompts, interface guidance, and situational questions and answers, to help users express their intentions more accurately. At the same time, the remote control system can also actively ask the user for some details or contextual information, such as "What aspect are you dissatisfied with?" "How much do you want to adjust?", etc., to collect more clues that help understand user needs. When processing the second user's voice, the remote control system usually needs to associate it with previous user commands and scene information for analysis to accurately grasp the changes and continuation of user needs. This may involve some complex semantic understanding and contextual reasoning technologies, such as reference resolution, omission recovery, and multi-round dialogue management. For example, when a user says "brighter" after dimming the light, the remote control system needs to understand the semantics of "more" and "some" and make appropriate incremental adjustments based on the previous brightness parameters. For another example, when a user says "set the wind speed to the lowest" after setting the air-conditioning temperature, the remote control system needs to determine whether the user is correcting the temperature instruction or setting a new wind speed instruction based on the context. In short, through the interactive link of the second user's voice input, the remote control system can better adapt to the dynamic changes in user needs and provide a more humane and intelligent service experience.
[0056] S108. If the user feedback is satisfactory, the current scene information, user commands, and device control instructions are recorded in association to update the multimodal scene understanding model and the user demand control instruction mapping table.
[0057] If the user feedback is satisfactory, it means that the execution result of the controlled device is good, and the remote control system can consider that the voice control process is successful and effective. At this time, the remote control system needs to record and save the complete interaction process for subsequent optimization and enhancement of the performance of the remote control system. Specifically, the remote control system will associate and store the three key elements of current scene information, user commands, and the final generated device control instructions to form a complete semantic mapping sample. Semantic mapping samples are crucial for improving speech understanding and command mapping models.
[0058] In the above embodiment, the remote control system combines user commands, current scene information, a multimodal scene understanding model, and a user demand control instruction mapping table to achieve accurate understanding and response to the first user's voice command. After determining the user's needs, the remote control system sends the corresponding device control instructions to the controlled device to control the controlled device to perform operations. After that, the remote control system obtains user feedback. If the user feedback is not satisfactory, it means that the effect of the voice remote control is not what the user wants, then the remote control system will re-acquire the second user's voice and re-perform voice remote control; if the user feedback is satisfactory, it means that the effect of the voice remote control is what the user wants, then the remote control system dynamically adjusts the multimodal scene understanding model and the user demand control instruction mapping table through this voice remote control, thereby achieving adaptive learning and optimization of the voice remote control system. Compared with the prior art, this method makes full use of scene information and user feedback to improve the accuracy of voice remote control and user experience.
[0059] The following is a more detailed description of the process of the method provided by this implementation. Figure 2 , is another flow chart of the voice remote control method in the embodiment of the present application.
[0060] S201, obtaining a historical status record of a device, where the historical status record includes parameter setting status of the device under different scenario information; The remote control system needs to obtain the past working status records of each device in advance, that is, the historical status records, which reflect the parameter settings of the device under different scene information. For example, for a smart light bulb, the historical status record may include its brightness value in different scenes such as "before bedtime scene", "movie scene", "reading scene", etc., which are not limited here; for a smart air conditioner, the historical status record may include its temperature settings in time and weather such as "summer daytime", "winter morning", "nap time", etc., which are not limited here. The historical status record can come from the user's manual operation log or from the automatic status report of the device, which is not limited here.
[0061] S202, determining a common mode of the device according to the historical status record as a user's personalized preference; After obtaining the historical status record of the device, the remote control system can automatically extract the user's personalized preference for the common mode of the device through a certain statistical analysis algorithm, that is, the user's personalized preference. For example, if the remote control system determines that the median of the user's setting of the light bulb brightness in the bedtime scene is 40% within the preset time period, the frequency exceeds the first preset frequency threshold, then the remote control system can determine the common mode of the light bulb "40% light bulb brightness before bedtime" as a user personalized preference; if the remote control system determines that the user sets the air conditioner temperature to 22 degrees Celsius in the afternoon time period, the frequency exceeds the second preset frequency threshold, then the remote control system can determine the common mode of the air conditioner "afternoon temperature 22 degrees" as a user personalized preference. In addition to statistical numerical analysis, the remote control system can also use some machine learning algorithms, such as pattern clustering, association rule mining, etc., to discover more complex user personalized preferences in the status record, such as "movie lighting scene + air conditioner low wind" and other combination settings. It should be noted that the extraction results of user personalized preferences still need to be manually checked and verified as necessary to avoid unreasonable associations. In short, through intelligent analysis of historical status records, the remote control system can automatically derive personalized device usage preferences and provide data support for subsequent personalized services.
[0062] S203, performing voice recognition on the received first user voice, and obtaining a user command converted from voice form to text form; For details, please refer to step S101, which will not be described in detail here.
[0063] S204, obtaining current scene information; For details, please refer to step S102, which will not be described in detail here.
[0064] S205, inputting the user command and the current scene information into a preset multimodal scene understanding model to obtain the user demand, which includes the user intention, the operation object and the operation parameter; For details, please refer to step S103, which will not be described in detail here.
[0065] S206, calculating the matching degree between the user demand and the user's personalized preference; The remote control system matches and calculates the user control demand generated in step S205 with the personalized preferences extracted in step S202. The remote control system can represent both the user demand and the user personalized preferences as structured feature vectors, and then calculates the matching degree between the user demand structured feature vector and the user personalized preference structured feature vector, such as cosine similarity. For example, if the user demand is "increase the air conditioning temperature in the living room by 2 degrees", and the user personalized preference includes "set the air conditioning temperature in the living room to 24 degrees in the summer evening", then the matching degree between the two may be very high, because both involve fine-tuning the air conditioning in the living room to a comfortable temperature in similar scenarios. It should be noted that the matching degree calculation needs to comprehensively consider various aspects of user demand and user personalized preferences, such as scenarios, equipment, parameters, etc., and weight the key elements therein to obtain a more accurate similarity result.
[0066] S207, when the matching degree exceeds a preset matching degree threshold, asking the user whether to set the device to a common mode; The preset matching degree threshold can be set according to the actual situation. When the matching degree between the structured feature vector of user demand and the structured feature vector of user personalized preference exceeds the preset matching degree threshold, it is considered that the user demand and the user personalized preference have sufficient correlation, and the remote control system can actively ask the user whether the common mode in the user personalized preference needs to be applied. The inquiry method can be voice prompt, pop-up window reminder, etc. For example, when the user says "I want to watch a movie", the remote control system calculates that the matching degree between the user demand and the user personalized preference of "movie viewing mode" is very high, and then the user can be asked "Do you want to switch to your common movie viewing mode?" This active confirmation interaction can avoid the user from repeating the function parameters and improve the fluency of the interaction. It should be noted that the timing and frequency of the inquiry need to be set reasonably, and the user should not be disturbed too frequently. If the user repeatedly denies the prompt, the remote control system should also record it and dynamically adjust the confirmation interaction strategy.
[0067] S208: After receiving confirmation feedback from the user, execute an operation of setting the device to a common mode.
[0068] If the user gives positive confirmation feedback to the remote control system's inquiry, indicating that he agrees to set the device to the common mode in the user's personalized preferences, then the remote control system can directly call the device status or parameter configuration contained in the user's personalized preference mode, without the user having to perform tedious settings. For example, after confirming the viewing mode, the remote control system can automatically switch the TV to the specified theater effect, dim the lights, close the doors and windows, etc., allowing the user to directly enter the preset scene experience. This one-click setting operation based on the user's personalized preferences can greatly simplify the user's control process and provide a more humane and intelligent voice interaction service. Of course, before executing the settings, the remote control system still needs to conduct necessary assessments on the execution safety to avoid negative impacts on users or the environment.
[0069] S209, based on a preset user demand control instruction mapping table, obtaining a device control instruction corresponding to the user command according to the user demand, wherein the user demand control instruction mapping table includes a correspondence between the user demand and the control instruction; For details, please refer to step S104, which will not be described in detail here.
[0070] S210, sending the device control instruction to the corresponding controlled device to control the controlled device to perform an operation; For details, please refer to step S105, which will not be described in detail here.
[0071] S211, sending the received execution result of the controlled device to the user and obtaining user feedback on the execution result; For details, please refer to step S106, which will not be described in detail here.
[0072] S212, recording user feedback and corresponding device control instructions to build a user satisfaction database; The remote control system will record the user feedback and the device control instructions in correspondence to build a user satisfaction database. User feedback here can be in various forms, such as voice ratings. The remote control system needs to define a quantitative indicator of satisfaction, such as a 1-5 point rating system. For example, when the user gives a 4-point satisfaction rating to the feedback after the air conditioner cools down, the remote control system records the correspondence between "the air conditioner lowers the temperature by 2 degrees" and "4-point satisfaction". By accumulating a large number of such corresponding samples, the remote control system can obtain a user satisfaction database that reflects the user's subjective feelings. It should be noted that user feedback needs to remove the user's personal privacy information and obtain the user's consent.
[0073] S213, regularly identifying the relationship between device control instructions and user feedback to obtain analysis results; After accumulating enough data, the remote control system can regularly analyze the user satisfaction database to identify the internal connection or pattern between device control instructions and user feedback, and obtain some optimized analysis results. For example, some statistical or data mining algorithms can be used to find out which device control instructions often correspond to higher or lower user satisfaction, which will help to control and adjust the device in the future. For example, when the temperature adjustment range is 2-3 degrees, the user feedback is the best, and the user satisfaction decreases beyond this range. Therefore, when the user needs to adjust the temperature, the remote control system can adjust the temperature within the range of 2-3 degrees.
[0074] S214. Adjust the user satisfaction database based on the analysis results.
[0075] After obtaining the analysis results, the remote control system strengthens the device control instructions with higher user satisfaction. In the user demand control instruction mapping table, the priority of these device control instructions is increased to make them easier to be selected in the subsequent instruction matching process. According to the analysis results, the device control instructions with lower user satisfaction are weakened or modified. In the user demand control instruction mapping table, the priority of these device control instructions is lowered, or unreasonable device control instructions are directly deleted to make them less likely to be matched and used. It is also possible to optimize the device control instructions by modifying the parameter range to make it more in line with user needs. At the same time, based on the above, the remote control system also adjusts and modifies the device control instructions and user satisfaction in the user satisfaction database accordingly.
[0076] In the above embodiment, the remote control system calculates the matching degree between the user's needs and the user's preferences. When the matching degree exceeds the preset matching degree threshold, the remote control system will actively initiate a confirmation inquiry to the user, prompting the user whether the device needs to be set to the common mode. This intelligent inquiry mechanism is a kind of prediction and completion of the user's intention, helping the user to omit the step of explicit command. The user only needs to make a simple positive feedback to the prompt of the remote control system, and then he can open the personalized scene with one click, and quickly configure the device to the most familiar and favorite state, so as to obtain a more intuitive and smooth control experience. This human-computer interaction mode breaks through the passive response limitations of traditional voice control. The system actively asks questions to the user and guides the user to confirm feedback, thereby realizing a hybrid active interaction form. The remote control system intervenes in the user's control process in a timely manner, and provides accurate control suggestions based on the user's previous behavior habits, which reduces the user's cognitive and operational burden while also increasing the user's sense of use.
[0077] The remote control system in the embodiment of the present invention is described below from the perspective of hardware processing. Figure 3 , is a schematic diagram of a physical device structure of a remote control system in an embodiment of the present application.
[0078] It should be noted that Figure 3 The structure of the remote control system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0079] like Figure 3 As shown, the remote control system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302 and the RAM 303 are connected to each other through the bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.
[0080] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, a button switch, etc.; an output section 307 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.
[0081] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.
[0082] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.
[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.
[0084] Specifically, the remote control system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the voice remote control method provided in the above embodiment is implemented.
[0085] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the remote control system described in the above embodiment; or may exist independently without being assembled into the remote control system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the remote control system, the remote control system implements the voice remote control method provided in the above embodiment.
[0086] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0087] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.
[0088] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
Claims
1. A voice remote control method, characterized in that: Applied to a remote control system, the method comprises: Performing speech recognition on the received first user speech to obtain a user command converted from speech form to text form; Get current scene information; Inputting the user command and the current scene information into a preset multimodal scene understanding model to obtain user needs, where the user needs include user intention, operation object, and operation parameters; Based on a preset user demand control instruction mapping table, a device control instruction corresponding to the user command is obtained according to the user demand, wherein the user demand control instruction mapping table includes a correspondence between the user demand and the control instruction; Sending the device control instruction to the corresponding controlled device to control the controlled device to perform an operation; Sending the received execution result of the controlled device to a user and obtaining user feedback from the user on the execution result; If the user feedback is dissatisfied, prompting the user to input a second user voice; If the user feedback is satisfactory, the current scene information, the user command and the device control instruction are recorded in association to update the multimodal scene understanding model and the user demand control instruction mapping table.
2. The method according to claim 1, characterized in that The user command and the current scene information are input into a preset multimodal scene understanding model to obtain user requirements, wherein the user requirements include user intention, operation object, and operation parameters, specifically including: Converting the user command and the current scene information into a user command vector and a current scene vector; Connecting the user command vector with the current scene vector to obtain a command-scene joint vector; The command-scenario joint vector is input into the multimodal scene understanding model to obtain a structured representation of user requirements as the user requirements, wherein the user requirements representation includes the user intention, the operation object, and the operation parameter.
3. The method according to claim 1, characterized in that The preset user demand control instruction mapping table is based on which the device control instruction corresponding to the user command is obtained according to the user demand. The user demand control instruction mapping table includes the corresponding relationship between the user demand and the control instruction, specifically including: Inputting the user intention, the operation object and the operation parameter into a preset instruction generation network; Based on the user demand control instruction mapping table, the device control instruction corresponding to the user command is generated through the instruction generation network.
4. The method according to claim 1, characterized in that The preset user demand control instruction mapping table is based on which the device control instruction corresponding to the user command is obtained according to the user demand. The user demand control instruction mapping table includes the corresponding relationship between the user demand and the control instruction, specifically including: Determine candidate devices related to the current scene information in a preset device association map, wherein the device association map includes each device node and its functional attributes, and association relationships between the device nodes; Matching the user intention and the operation object with the candidate devices and their corresponding functional attributes to determine a controlled device; According to the controlled device and the operating parameters, the device control instruction corresponding to the user command is matched from the user demand control instruction mapping table.
5. The method according to claim 1, characterized in that Before the step of performing speech recognition on the received first user speech to obtain a user command converted from speech form to text form, the method further includes: Obtaining a historical status record of a device, wherein the historical status record includes parameter setting status of the device under different scenario information; A common mode of the device is determined according to the historical status record as a user personalized preference.
6. The method according to claim 5, characterized in that After the step of inputting the user command and the current scene information into a preset multimodal scene understanding model to obtain a user demand, wherein the user demand includes a user intention, an operation object, and an operation parameter, the method further includes: Calculating the matching degree between the user demand and the user personalized preference; When the matching degree exceeds a preset matching degree threshold, asking the user whether to set the device to the common mode; After receiving the confirmation feedback from the user, an operation of setting the device to the common mode is performed.
7. The method according to claim 1, characterized in that After the step of sending the received execution result of the controlled device to the user and obtaining user feedback from the user on the execution result, the method further includes: Recording the user feedback and corresponding device control instructions to build a user satisfaction database; Regularly identifying the relationship between the device control instruction and the user feedback to obtain an analysis result; The user satisfaction database is adjusted based on the analysis results.
8. A remote control system, characterized in that: The remote control system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the remote control system to execute the method described in any one of claims 1-7.
9. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the remote control system, the remote control system is caused to execute the method according to any one of claims 1 to 7.
10. A computer program product, characterized in that When the computer program product is run on a remote control system, the remote control system is caused to execute the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Wakeup-free voice interaction method and device, equipment and storage medium
CN109326289A
Voice control method and device, storage medium and air conditioner
CN109671427A
Method based on voice control equipment and control terminal equipment
CN111308921A
Webpage classification and identification method and device
CN113222022A
Control instruction prediction method and device, storage medium and electronic device
CN114755931A
Cited By
Model training method, voice interaction method, server and storage medium
CN119763552A
AI voice processing method and system based on intelligent device
CN119920256A
Whole house intelligent scene generation system based on natural language
CN120010282A
Automatic adaptive packaging system based on voice control packaging equipment
CN120431928A
Automatically adapt the packaging system to the packaging equipment based on voice control
CN120431928B
