A smart home device control method and device, electronic device, and storage medium

Abstract

The embodiment of the present invention provides a method and apparatus for controlling smart home devices, an electronic device, and a storage medium, which obtains the user's input command data and the status information data of the smart home device; and performs residual constraint processing on the input command data and the status information data to generate a fusion vector; thereby determining the target smart home device that the user intends to control according to the fusion vector and generating a control command; and then controlling the target smart home device according to the control command. The method realizes unified control of the target smart home device that the user intends to control by generating a unified control command in response to the user's input command, reduces the error rate of user intention recognition, improves the accuracy of the control command, and improves the response efficiency and user satisfaction of the smart home system.

Description

Technical Field

[0001] The present invention relates to the field of smart home technology, and in particular to a smart home device control method and device, electronic equipment, and storage medium. Background Art

[0002] With the rapid development of information technology and the Internet of Things, smart home systems have become part of modern life. These smart home systems usually include a variety of smart devices, such as smart lamps, thermostats, security cameras, etc., designed to improve living comfort, safety and energy efficiency. Users can interact with these devices by inputting commands in a variety of ways.

[0003] However, although smart home devices provide many conveniences, smart home devices of different brands often use different communication protocols and control interfaces, which makes users face compatibility and operational complexity issues when using multi-brand devices. It is impossible to uniformly control multi-brand smart home devices, which limits the flexibility of user interaction and the intelligent response capabilities of the system. Summary of the invention

[0004] In view of the above problems, a method and device for controlling a smart home device, an electronic device, and a storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A method for controlling a smart home device, the method comprising:

[0006] Acquire the user's input command data and the status information data of the smart home device;

[0007] Performing residual constraint processing on the input instruction data and the state information data to generate a fusion vector;

[0008] Determining, according to the fusion vector, a target smart home device that the user intends to control, and generating a control instruction;

[0009] Controlling the target smart home device according to the control instruction;

[0010] The input command data includes voice command data and gesture command data, and the performing residual constraint processing on the input command data and the state information data to generate a fusion vector includes:

[0011] Encoding the voice command data, gesture command data and status information data to generate a first encoding vector, a second encoding vector and a third encoding vector;

[0012] performing feature enhancement processing and codebook constraint processing on the first coding vector, the second coding vector and the third coding vector in sequence to generate a fourth coding vector, a fifth coding vector and a sixth coding vector;

[0013] The fourth coding vector, the fifth coding vector and the sixth coding vector are weighted averaged to generate the fusion vector.

[0014] Optionally, controlling the target smart home device according to the control instruction includes:

[0015] Performing protocol conversion processing on the control instruction to generate a control protocol recognizable by the target smart home device;

[0016] The control protocol is sent to the target smart home device to control the target smart home device.

[0017] Optionally, the method further includes:

[0018] Receiving status feedback information generated by the target smart home device according to the control protocol, and determining whether the control instruction has been correctly executed by the target smart home device according to the status feedback information;

[0019] If the control instruction is not correctly executed by the target smart home device, the control protocol is sent to the target smart home device again and / or the user is notified to perform a status check on the target smart home device.

[0020] Optionally, determining, according to the fusion vector, a target smart home device that the user intends to control, and generating a control instruction includes:

[0021] The fusion vector is decoded to determine the target smart home device that the user intends to control and generate the control instruction.

[0022] Optionally, before performing residual constraint processing on the input instruction data and the state information data, the method further includes:

[0023] The input instruction data and the status information data are preprocessed.

[0024] A smart home device control device, the device comprising:

[0025] A data acquisition module, used to acquire the user's input command data and the status information data of the smart home device;

[0026] A fusion vector generation module, used for performing residual constraint processing on the input instruction data and the state information data to generate a fusion vector;

[0027] A control instruction generation module, used to determine the target smart home device that the user intends to control according to the fusion vector, and generate a control instruction;

[0028] A device control module, used to control the target smart home device according to the control instruction;

[0029] The input command data includes voice command data and gesture command data, and the fusion vector generation module includes:

[0030] The encoding processing submodule is used to encode the voice command data, gesture command data and status information data to generate a first encoding vector, a second encoding vector and a third encoding vector;

[0031] A feature enhancement processing and codebook constraint processing submodule, used to perform feature enhancement processing and codebook constraint processing on the first coding vector, the second coding vector and the third coding vector in sequence to generate a fourth coding vector, a fifth coding vector and a sixth coding vector;

[0032] The weighted average processing submodule is used to perform weighted average processing on the fourth coding vector, the fifth coding vector and the sixth coding vector to generate the fusion vector.

[0033] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the smart home device control method as described above when executed by the processor.

[0034] A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the smart home device control method as described above is implemented.

[0035] A computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the smart home device control method as described above.

[0036] The embodiments of the present invention have the following advantages:

[0037] The present invention provides a method for controlling smart home devices, which obtains the user's input command data and the state information data of the smart home devices; performs residual constraint processing on the input command data and the state information data to generate a fusion vector; thereby determining the target smart home device that the user intends to control according to the fusion vector and generating a control command; and then controlling the target smart home device according to the control command. The method realizes unified control of the target smart home devices that the user intends to control by generating a unified control command in response to the user's input command, reduces the error rate of user intention recognition, improves the accuracy of the control command, and improves the response efficiency and user satisfaction of the smart home system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0039] Figure 1 is a flowchart of a method for controlling a smart home device provided by some embodiments of the present invention;

[0040] Figure 2 It is a schematic diagram of the overall execution flow of the present invention provided by some embodiments of the present invention;

[0041] Figure 3 It is a structural schematic diagram of a smart home device control device provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] In the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. This is only for the purpose of clearly describing the technical solutions of the embodiments of the present application, and shall not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0044] In the embodiments of the present application, the terms "upper", "lower", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0045] With the rapid development of information technology and the Internet of Things, smart home systems have become part of modern life. These smart home systems usually include a variety of smart devices, such as smart lamps, thermostats, security cameras, etc., designed to improve living comfort, safety and energy efficiency. Users can interact with these devices by inputting commands in a variety of ways.

[0046] However, although smart home devices provide many conveniences, smart home devices of different brands often use different communication protocols and control interfaces, which makes users face compatibility and operational complexity issues when using multi-brand devices. It is impossible to uniformly control multi-brand smart home devices, which limits the flexibility of user interaction and the intelligent response capabilities of the system.

[0047] In order to solve the above problems, the present invention improves the smart home device control method in the related art based on the core technical concept of generating a unified control instruction in response to the user's input instruction to uniformly control the target smart home device that the user intends to control. The present invention will be described in detail in conjunction with the accompanying drawings below:

[0048] Reference Figure 1 , shows a flowchart of a method for controlling a smart home device provided by some embodiments of the present invention, which may specifically include the following steps:

[0049] Step 101, obtaining the user's input command data and the status information data of the smart home device;

[0050] In a specific implementation, a smart home device control system can be set up and a multimodal perception module can be set up in the system to obtain the user's input command data and the status information data of the smart home device. The user's input command data may include voice command data and gesture command data. Specifically, for voice command data, the voice command issued by the user can be captured by a high-quality microphone array. The collected voice signal can be preprocessed (such as noise filtering and enhancement), and then the voice signal can be converted into a text command using voice recognition technology (such as Google ASR or Amazon Alexa), so that the system can recognize the command issued by the user, such as "turn on the living room light."

[0051] Furthermore, for gesture command data, the user's gestures can be collected through high-definition cameras or other sensors. The collected gesture signals can be pre-processed (such as image enhancement, background removal, and gesture segmentation), and then gesture recognition can be performed using deep learning technology (such as convolutional neural network CNN). For example, the system can recognize the user's gesture action of turning on and off the light.

[0052] Furthermore, for the status information data of smart home devices, the current status information of smart home devices can be obtained through the API interface or sensor of the smart device, such as temperature, humidity, light intensity, and switch status information, etc. After preprocessing, these data can be converted into standardized state vectors for subsequent system processing and decision-making.

[0053] Step 102, performing residual constraint processing on the input instruction data and the state information data to generate a fusion vector;

[0054] The input command data includes voice command data and gesture command data, and the performing residual constraint processing on the input command data and the state information data to generate a fusion vector includes:

[0055] Encoding the voice command data, gesture command data and status information data to generate a first encoding vector, a second encoding vector and a third encoding vector;

[0056] performing feature enhancement processing and codebook constraint processing on the first coding vector, the second coding vector and the third coding vector in sequence to generate a fourth coding vector, a fifth coding vector and a sixth coding vector;

[0057] Performing weighted average processing on the fourth coding vector, the fifth coding vector and the sixth coding vector to generate the fusion vector;

[0058] In the specific implementation, a residual constrained codeword network (RCQN) can be designed to implement residual constraint processing of input command data and state information data. Specifically, an independent encoder can be designed for each modality to encode speech, gesture and state information into low-dimensional vectors for representation, for example:

[0059] Speech encoder: A convolutional neural network (CNN) and a long short-term memory network (LSTM) architecture can be used to extract the time-frequency features and time series dependencies of the speech signal to encode the speech command data into a low-dimensional vector (i.e., the first encoding vector).

[0060] Gesture encoder: A pre-trained convolutional neural network (such as ResNet) can be used to extract the features of the gesture image, and then the feature vector is reduced in dimension through a fully connected layer to encode the gesture instruction data into a low-dimensional vector (i.e., the second encoded vector).

[0061] Device state encoder: A multi-layer perceptron (MLP) can be used to encode the device state information into a low-dimensional vector (i.e., the third encoded vector).

[0062] Furthermore, multiple residual blocks can be added after each encoder to further extract features (i.e., feature enhancement processing). Residual blocks can solve the gradient vanishing problem in deep network training through skip connections.

[0063] Specifically, the basic formula of the residual block is:

[0064]

[0065] Among them, Input is the input vector, Output is the output, is a vector obtained by performing nonlinear transformation on the Input through several fully connected layers, + means concatenating two vectors, and ReLU is the activation function.

[0066] Furthermore, a shared codebook vector space can be introduced, and constraints can be introduced during the network training process so that the encoding vectors of different modes are close to the codebook vector layer by layer (i.e., codebook constraint processing), for example, including:

[0067] Codebook matrix: A fixed-size codebook matrix C can be set, where each column is a codebook vector. This matrix can be used as a learnable parameter during training.

[0068] Codebook constraint loss: A constraint term is added to the loss function to make the encoding vector of each mode as close to the codebook vector as possible. The form of the loss function can be:

[0069]

[0070] Among them, zi represents the i-th encoding vector and cj represents the j-th codebook vector.

[0071] The total loss function can be:

[0072]

[0073] Among them, α is the weight coefficient used to balance the task loss and the codebook constraint loss, Ltask is the task loss, and Lcodebook is the codebook constraint function. By balancing the task loss and the codebook constraint loss through the α weight coefficient, the total loss L can be obtained.

[0074] Specifically, as described in the above steps, the first coding vector, the second coding vector and the third coding vector can be subjected to feature enhancement processing and codebook constraint processing in sequence to generate a fourth coding vector (corresponding to the first coding vector), a fifth coding vector (corresponding to the second coding vector) and a sixth coding vector (corresponding to the third coding vector).

[0075] Furthermore, the coding vectors of each mode may be fused in the codebook space to form a unified fusion vector representation (i.e., weighted average processing), which may include, for example:

[0076] Weighted average: The encoding vectors are weighted averaged according to the weights of the modalities to form a fusion vector. For example, the following formula can be used:

[0077]

[0078] Among them, ω1, ω2, ω3 are weight coefficients, and ω1+ω2+ω3=1; Zspeech is the speech modality coding vector (i.e., the fourth coding vector), Zgesture is the gesture modality coding vector (i.e., the fifth coding vector), and Zstate is the state modality coding vector (i.e., the sixth coding vector). Zfusion is the fusion vector obtained by weighted averaging.

[0079] Furthermore, an attention mechanism can be introduced, that is, by calculating the attention weights, the encoding vectors of different modalities are weighted summed.

[0080]

[0081] Among them, αi is the weight calculated by the attention mechanism, satisfying .

[0082] In some embodiments of the present invention, before performing residual constraint processing on the input instruction data and the state information data, the method further includes:

[0083] The input instruction data and the status information data are preprocessed.

[0084] In practical applications, the input command data and the status information data can be preprocessed. Specifically, the voice command data can be preprocessed, for example, noise filtering and enhancement can be performed to eliminate background noise and improve voice quality. Then, voice recognition technology can be used to convert the processed voice signal into recognizable text commands to ensure that the system accurately understands the user's voice input.

[0085] Furthermore, gesture preprocessing can be performed on the gesture command data. For example, the image data captured by the camera can be processed through image processing steps including denoising, contrast enhancement, and background removal to improve the accuracy of gesture recognition. Subsequently, a deep learning model can be used to extract features and perform gesture recognition on the processed image to convert the user's gesture movements into commands that the system can understand.

[0086] Furthermore, the state information data of smart home devices can be pre-processed, for example, the original state data of smart devices can be standardized and converted into a unified state vector format. This ensures that the state information of different devices can be uniformly managed and compared in the system, improving the system's response speed and accuracy to device state changes.

[0087] Step 103, determining the target smart home device that the user intends to control according to the fusion vector, and generating a control instruction;

[0088] In the specific implementation, a corresponding decoder can be set and the fusion vector can be input into the decoder to generate the final control instruction. The decoder can adopt RNN or Transformer structure to generate natural language instructions. Specifically:

[0089] RNN decoder: suitable for processing time series data and can generate natural language instructions.

[0090] Transformer decoder: It has parallel computing capabilities and long dependency modeling capabilities. It can process inputs through the self-attention mechanism and generate control instructions. It can be selected according to actual needs.

[0091] Furthermore, the fused vector can be parsed to determine the user's intention and the target smart home device. Specifically, for user intention recognition, the user's intention can be identified by analyzing the fused feature vector; for example, it can be determined whether the user wants to turn on the lights, adjust the temperature, or play music. For target smart home device determination, the target smart home device to be controlled can be determined based on the user's intention and the current device status. For example, if the user's intention is to turn on the lights, the system will identify the specific lighting device to be controlled (such as a living room light or a bedroom light).

[0092] Furthermore, specific control instructions can be generated based on the analysis results, and the control protocols of smart home devices of multiple brands can be supported. For example, when generating the instruction "turn on the living room lights", the instruction can be formatted into corresponding control commands according to the protocols of devices of different brands. When generating control instructions, appropriate adjustments can be made according to the control protocols and interfaces of smart home devices of different brands to ensure that the instructions can be correctly executed by the target device.

[0093] Furthermore, instructions can be optimized according to the characteristics of different smart home devices to improve control accuracy and execution efficiency, which can specifically include:

[0094] Command calibration: Calibrate control commands based on the characteristics of different smart home devices. For example, when adjusting the temperature of an air conditioner, the control command can be optimized based on the temperature adjustment accuracy of the smart home device.

[0095] Execution efficiency optimization: The instructions can be optimized according to the response speed and current status of smart home devices to improve the accuracy and execution efficiency of control. For example, the brightness of lights can be adjusted in segments to prevent flickering.

[0096] In some embodiments of the present invention, determining the target smart home device that the user intends to control according to the fusion vector and generating a control instruction includes:

[0097] The fusion vector is decoded to determine the target smart home device that the user intends to control and generate the control instruction.

[0098] In a specific implementation, a corresponding decoder may be set and the fused vector may be input into the decoder to generate a final control instruction. The decoder may adopt an RNN or Transformer structure to generate natural language instructions.

[0099] Specifically, the fused vector can be parsed during the decoding process to determine the user's intention and the target smart home device. For example, for user intention recognition, the user's intention can be identified by analyzing the fused feature vector; for example, it can be determined whether the user wants to turn on the lights, adjust the temperature, or play music. For target smart home device determination, the target smart home device to be controlled can be determined based on the user's intention and the current device status. For example, if the user's intention is to turn on the lights, the system will identify the specific lighting device to be controlled (such as a living room light or a bedroom light).

[0100] Furthermore, specific control instructions can be generated based on the analysis results, and the control protocols of smart home devices of multiple brands can be supported. For example, when generating the instruction "turn on the living room lights", the instruction can be formatted into corresponding control commands according to the protocols of devices of different brands. When generating control instructions, appropriate adjustments can be made according to the control protocols and interfaces of smart home devices of different brands to ensure that the instructions can be correctly executed by the target device.

[0101] Furthermore, instructions can be optimized according to the characteristics of different smart home devices to improve control accuracy and execution efficiency, which can specifically include:

[0102] Command calibration: Calibrate control commands based on the characteristics of different smart home devices. For example, when adjusting the temperature of an air conditioner, the control command can be optimized based on the temperature adjustment accuracy of the smart home device.

[0103] Execution efficiency optimization: The instructions can be optimized according to the response speed and current status of smart home devices to improve the accuracy and execution efficiency of control. For example, the brightness of lights can be adjusted in segments to prevent flickering.

[0104] Step 104: Control the target smart home device according to the control instruction.

[0105] In practical applications, the generated standardized control instructions can be converted into control protocols that can be recognized by smart home devices, for example, including:

[0106] Standardized instruction format: Convert the generated control instructions into a standardized format to ensure the universality of the instructions.

[0107] Device protocol conversion: Convert standardized control instructions into control protocols that target smart home devices can recognize. For example, convert the standardized "turn on the light" instruction into the control protocol format of a specific brand of smart lights.

[0108] Furthermore, instructions may be sent to the target smart home device through the device's API or control interface, and may include, for example:

[0109] API interface call: send control instructions to smart home devices through the target device's API interface. For example, call the smart light control API to send a light-on instruction.

[0110] Network communication: Communication technologies such as Wi-Fi, Zigbee or Bluetooth can be used to send control commands to target smart home devices through the home network.

[0111] Furthermore, it is also possible to receive status feedback information from smart home devices, update the system status, and perform subsequent control according to actual needs, which may specifically include:

[0112] Feedback reception: Receive status feedback from smart home devices to confirm whether the instructions are correctly executed by the smart home devices. For example, receive the current brightness status of smart lights.

[0113] System status update: Update the status of the smart home device control system according to the feedback information of the smart home device to ensure that the status of the smart home device control system is synchronized with the actual smart home device status.

[0114] Subsequent control: Perform necessary subsequent control based on the feedback from smart home devices. For example, if the light fails to turn on, the smart home device control system will resend the command or prompt the user to check the device status.

[0115] Furthermore, in order to ensure the accuracy and robustness of the smart home device control system, the residual constraint codebook network can be trained and optimized, which may specifically include:

[0116] Data Collection: Collect a large amount of multimodal data to cover different scenarios and smart home devices.

[0117] Model training: supervised learning methods are used to train RCQN using the collected data. Transfer learning and data augmentation techniques can be used to improve the generalization ability of the model.

[0118] Model optimization: Improve the performance of the model by adjusting parameters and optimizing the architecture. Cross-validation and test sets can be used to evaluate the accuracy and robustness of the model.

[0119] Online learning: In actual use, the smart home device control system can continuously optimize the model through online learning to adapt to the user's personalized needs and new smart home devices.

[0120] In some embodiments of the present invention, controlling the target smart home device according to the control instruction includes:

[0121] Performing protocol conversion processing on the control instruction to generate a control protocol recognizable by the target smart home device;

[0122] The control protocol is sent to the target smart home device to control the target smart home device.

[0123] In a specific implementation, the generated standardized control instructions may be converted into a control protocol that can be recognized by the smart home device, for example, the following may be included:

[0124] Standardized instruction format: Convert the generated control instructions into a standardized format to ensure the universality of the instructions.

[0125] Device protocol conversion: Convert standardized control instructions into control protocols that target smart home devices can recognize. For example, convert the standardized "turn on the light" instruction into the control protocol format of a specific brand of smart lights.

[0126] Furthermore, instructions may be sent to the target smart home device through the device's API or control interface, and may include, for example:

[0127] API interface call: send control instructions to smart home devices through the target device's API interface. For example, call the smart light control API to send a light-on instruction.

[0128] Network communication: Communication technologies such as Wi-Fi, Zigbee or Bluetooth can be used to send control commands to target smart home devices through the home network.

[0129] In some embodiments of the present invention, the method further comprises:

[0130] Receiving status feedback information generated by the target smart home device according to the control protocol, and determining whether the control instruction has been correctly executed by the target smart home device according to the status feedback information;

[0131] If the control instruction is not correctly executed by the target smart home device, the control protocol is sent to the target smart home device again and / or the user is notified to perform a status check on the target smart home device.

[0132] In actual applications, you can receive status feedback information from smart home devices, update the system status, and perform subsequent control according to actual needs, which may include:

[0133] Feedback reception: Receive status feedback from smart home devices to confirm whether the instructions are correctly executed by the smart home devices. For example, receive the current brightness status of smart lights.

[0134] System status update: Update the status of the smart home device control system according to the feedback information of the smart home device to ensure that the status of the smart home device control system is synchronized with the actual smart home device status.

[0135] Subsequent control: Perform necessary subsequent control based on the feedback from smart home devices. For example, if the light fails to turn on, the smart home device control system will resend the command or prompt the user to check the device status.

[0136] The following will be combined Figure 2 The embodiments of the present invention are further described as follows:

[0137] Reference Figure 2 The overall execution process of the smart home device control method provided by the present invention can be summarized as follows:

[0138] Step 1: Collect voice modal data, gesture modal data and device status modal data respectively, and perform corresponding voice data preprocessing (voice denoising, language recognition, etc.), gesture data preprocessing (image denoising, enhancement, etc.) and status data preprocessing (standardization processing).

[0139] Step 2: residual constrained codebook network (RCQN) design, that is, performing residual constraint processing on the preprocessed speech data, gesture data and state data through the residual constrained codebook network to generate a fusion vector.

[0140] Step three, decision control, that is, determining the target smart home device that the user intends to control according to the fusion vector, and generating a control instruction;

[0141] Step 4: command execution, that is, controlling the target smart home device according to the control command.

[0142] Step 5: System training and optimization. In order to ensure the accuracy and robustness of the smart home device control system, the residual constraint codebook network can be trained and optimized, which may include:

[0143] Data Collection: Collect a large amount of multimodal data to cover different scenarios and smart home devices.

[0144] Model training: supervised learning methods are used to train RCQN using the collected data. Transfer learning and data augmentation techniques can be used to improve the generalization ability of the model.

[0145] Model optimization: Improve the performance of the model by adjusting parameters and optimizing the architecture. Cross-validation and test sets can be used to evaluate the accuracy and robustness of the model.

[0146] Online learning: In actual use, the smart home device control system can continuously optimize the model through online learning to adapt to the user's personalized needs and new smart home devices.

[0147] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0148] Reference Figure 3 , shows a schematic diagram of the structure of a smart home device control device provided by some embodiments of the present invention; specifically, it may include the following modules:

[0149] The data acquisition module 301 is used to acquire the user's input command data and the status information data of the smart home device;

[0150] A fusion vector generating module 302 is used to perform residual constraint processing on the input instruction data and the state information data to generate a fusion vector;

[0151] A control instruction generating module 303 is used to determine the target smart home device that the user intends to control according to the fusion vector, and generate a control instruction;

[0152] The device control module 304 is used to control the target smart home device according to the control instruction;

[0153] The input command data includes voice command data and gesture command data, and the fusion vector generation module 302 includes:

[0154] The encoding processing submodule is used to encode the voice command data, gesture command data and status information data to generate a first encoding vector, a second encoding vector and a third encoding vector;

[0155] A feature enhancement processing and codebook constraint processing submodule, used to perform feature enhancement processing and codebook constraint processing on the first coding vector, the second coding vector and the third coding vector in sequence to generate a fourth coding vector, a fifth coding vector and a sixth coding vector;

[0156] The weighted average processing submodule is used to perform weighted average processing on the fourth coding vector, the fifth coding vector and the sixth coding vector to generate the fusion vector.

[0157] In some embodiments of the present invention, the device control module 304 includes:

[0158] A protocol conversion processing submodule, used to perform protocol conversion processing on the control instruction to generate a control protocol recognizable by the target smart home device;

[0159] The device control submodule is used to send the control protocol to the target smart home device to control the target smart home device.

[0160] In some embodiments of the present invention, the device further comprises:

[0161] A status feedback information receiving module, used to receive status feedback information generated by the target smart home device according to the control protocol, and determine whether the control instruction has been correctly executed by the target smart home device according to the status feedback information;

[0162] The control instruction incorrect execution processing module is used to send the control protocol to the target smart home device again and / or notify the user to perform a status check on the target smart home device if the control instruction is not correctly executed by the target smart home device.

[0163] In some embodiments of the present invention, the control instruction generating module 303 includes:

[0164] The decoding processing submodule is used to decode the fusion vector to determine the target smart home device that the user intends to control and generate the control instruction.

[0165] In some embodiments of the present invention, the device further comprises:

[0166] A preprocessing module is used to preprocess the input instruction data and the status information data.

[0167] Some embodiments of the present invention further provide an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the above-mentioned smart home device control method is implemented.

[0168] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned smart home device control method is implemented.

[0169] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above smart home device control method when executed by a processor.

[0170] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0172] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0173] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0177] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0178] The above provides a detailed introduction to a smart home device control method and device, an electronic device, and a storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for controlling smart home devices, It is characterized in that The method comprises: Acquire the user's input command data and the status information data of the smart home device; Perform residual constraint processing on the input instruction data and the state information data through a residual constraint codebook network to generate a fusion vector; Determining, according to the fusion vector, a target smart home device that the user intends to control, and generating a control instruction; Controlling the target smart home device according to the control instruction; The input command data includes voice command data and gesture command data, and the residual constraint processing is performed on the input command data and the state information data through the residual constraint codebook network to generate a fusion vector, including: Encoding the voice command data, gesture command data and status information data to generate a first encoding vector, a second encoding vector and a third encoding vector; performing feature enhancement processing and codebook constraint processing on the first coding vector, the second coding vector and the third coding vector in sequence to generate a fourth coding vector, a fifth coding vector and a sixth coding vector; Performing weighted average processing on the fourth coding vector, the fifth coding vector and the sixth coding vector to generate the fusion vector; The step of sequentially performing feature enhancement processing and codebook constraint processing on the first coding vector, the second coding vector, and the third coding vector to generate a fourth coding vector, a fifth coding vector, and a sixth coding vector includes: Performing the feature enhancement processing on the first coding vector, the second coding vector and the third coding vector through a plurality of residual blocks; By introducing a shared codebook vector space and introducing constraints during the network training process, the codebook constraint processing is performed on the first code vector, the second code vector and the third code vector that have completed the feature enhancement processing, so as to generate the fourth code vector, the fifth code vector and the sixth code vector; The step of performing weighted average processing on the fourth coding vector, the fifth coding vector and the sixth coding vector to generate the fusion vector includes: The fourth coding vector, the fifth coding vector and the sixth coding vector are weighted averaged in the codebook vector space to generate the fusion vector.

2. The method according to claim 1, It is characterized in that The controlling the target smart home device according to the control instruction includes: Performing protocol conversion processing on the control instruction to generate a control protocol recognizable by the target smart home device; The control protocol is sent to the target smart home device to control the target smart home device.

3. The method according to claim 2, It is characterized in that The method further comprises: Receiving status feedback information generated by the target smart home device according to the control protocol, and determining whether the control instruction has been correctly executed by the target smart home device according to the status feedback information; If the control instruction is not correctly executed by the target smart home device, the control protocol is sent to the target smart home device again and / or the user is notified to perform a status check on the target smart home device.

4. The method according to claim 1, It is characterized in that The step of determining the target smart home device that the user intends to control according to the fusion vector and generating a control instruction includes: The fusion vector is decoded to determine the target smart home device that the user intends to control and generate the control instruction.

5. The method according to claim 1, It is characterized in that Before performing residual constraint processing on the input instruction data and the state information data, the method further includes: The input instruction data and the status information data are preprocessed.

6. A smart home equipment control device, It is characterized in that The device comprises: A data acquisition module, used to acquire the user's input command data and the status information data of the smart home device; A fusion vector generation module, used for performing residual constraint processing on the input instruction data and the state information data to generate a fusion vector; A control instruction generation module, used to determine the target smart home device that the user intends to control according to the fusion vector, and generate a control instruction; A device control module, used to control the target smart home device according to the control instruction; The input command data includes voice command data and gesture command data, and the fusion vector generation module includes: The encoding processing submodule is used to encode the voice command data, gesture command data and status information data to generate a first encoding vector, a second encoding vector and a third encoding vector; A feature enhancement processing and codebook constraint processing submodule, used to perform feature enhancement processing and codebook constraint processing on the first coding vector, the second coding vector and the third coding vector in sequence to generate a fourth coding vector, a fifth coding vector and a sixth coding vector; A weighted average processing submodule, used for performing weighted average processing on the fourth coding vector, the fifth coding vector and the sixth coding vector to generate the fusion vector; The step of sequentially performing feature enhancement processing and codebook constraint processing on the first coding vector, the second coding vector, and the third coding vector to generate a fourth coding vector, a fifth coding vector, and a sixth coding vector includes: Performing the feature enhancement processing on the first coding vector, the second coding vector and the third coding vector through a plurality of residual blocks; By introducing a shared codebook vector space and introducing constraints during the network training process, the codebook constraint processing is performed on the first code vector, the second code vector and the third code vector that have completed the feature enhancement processing, so as to generate the fourth code vector, the fifth code vector and the sixth code vector; The step of performing weighted average processing on the fourth coding vector, the fifth coding vector and the sixth coding vector to generate the fusion vector includes: The fourth coding vector, the fifth coding vector and the sixth coding vector are weighted averaged in the codebook vector space to generate the fusion vector.

7. An electronic device, It is characterized in that The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the smart home device control method as claimed in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the smart home device control method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the smart home device control method according to any one of claims 1 to 5.

Images