Smart home equipment control method, device, medium and system

Abstract

The invention provides a control method, device, medium and system for smart home equipment, and the method comprises the steps: carrying out the preprocessing of all types of data in multi-modal data, obtaining the preprocessing data, carrying out the feature extraction of the preprocessing data, obtaining the feature data, enabling the data after the feature extraction to be more accurate, and enabling the data to be more accurate. According to the embodiment of the invention, the feature data can represent the real intention of the user better, then the feature data is adopted to train the capsule network model to obtain the final capsule network model, so that compared with the existing scheme, the real intention of the user can be identified more accurately, and finally the final capsule network model is applied to perform corresponding control on the smart home equipment. Therefore, the problem that the smart home cannot meet the real demand of the user only through the instruction of one mode of voice in the existing scheme is solved.

Description

Technical Field

[0001] The present application relates to the technical field of smart home device control, and in particular to a control method, device, medium and system for smart home devices. Background Art

[0002] In the field of smart home technology, users generally use voice control to control various smart homes. However, as the functions of smart home products continue to increase, voice control alone sometimes cannot meet the real needs of users. In other words, voice commands alone cannot meet the real needs of users. This often reduces the user experience when using smart homes, thus affecting the reputation of the smart home.

[0003] That is, the existing solutions cannot enable smart homes to meet the real needs of users through voice commands alone. Summary of the invention

[0004] The main purpose of the present application is to provide a control method, device, medium and system for smart home devices, so as to at least solve the problem that the existing solutions cannot enable smart homes to meet the real needs of users through voice commands alone.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a control method for a smart home device is provided, the method comprising: acquiring multimodal data, the multimodal data representing data used to control the smart home device, the multimodal data including voice data, text data and image data of a user; preprocessing each type of data in the multimodal data to obtain preprocessed data, and performing feature extraction on the preprocessed data to obtain feature data; training a capsule network model using the feature data to obtain a final capsule network model; and applying the final capsule network model to control the smart home device accordingly.

[0006] Optionally, the preprocessed data includes first preprocessed data, second preprocessed data and third preprocessed data, and each type of data in the multimodal data is preprocessed to obtain preprocessed data, including: performing word segmentation and normalization processing on the text data in the multimodal data in turn to obtain the first preprocessed data; performing frame segmentation, windowing and normalization processing on the voice data in the multimodal data in turn to obtain the second preprocessed data; and performing cropping, scaling and normalization processing on the image data in the multimodal data in turn to obtain the third preprocessed data.

[0007] Optionally, the feature data includes first feature data, second feature data and third feature data, and feature extraction is performed on the preprocessed data to obtain the feature data, including: using the bag-of-words model technology to extract features from the first preprocessed data to obtain the first feature data; using short-time Fourier transform technology and Mel-frequency cepstral coefficient technology to extract features from the second preprocessed data to obtain the second feature data; using convolutional neural network technology to extract features from the third preprocessed data to obtain the third feature data.

[0008] Optionally, in the process of using the feature data to train the capsule network model to obtain the final capsule network model, the method also includes: dividing the feature data into a training set and a test set, and using a dynamic routing algorithm to enable the capsule network model to learn the relationship and pattern between different modal features in the training set; using a back propagation algorithm to update the weights of the capsule network model to minimize the loss value of the loss function.

[0009] Optionally, in the process of updating the weights of the capsule network model using a back propagation algorithm to minimize the loss function, the method further includes:

[0010] The loss function is constructed as:

[0011]

[0012] Among them, L is the loss value of the loss function, K is the total number of dimensions, and y i is the true label of the i-th dimension, p i is the predicted probability distribution of the current capsule network model in the i-th dimension.

[0013] Optionally, the feature data is used to train a capsule network model to obtain a final capsule network model, including:

[0014] Obtain the current precision and current recall of the current capsule network model;

[0015] according to Determine the model performance coefficient;

[0016] The model performance coefficient represents the harmonic mean of the current precision and the current recall, F1 is the model performance coefficient, Precision is the current precision, and Recall is the current recall;

[0017] When the model performance coefficient is less than the coefficient threshold, determining the current capsule network model as the final capsule network model;

[0018] When the model performance coefficient is greater than or equal to the coefficient threshold, prompt information is generated to prompt that at least the structure of the current capsule network model needs to be adjusted.

[0019] Optionally, applying the final capsule network model to control the smart home device accordingly includes: acquiring current multimodal data; using the final capsule network model to process the current multimodal data to obtain a control scheme; and using the control scheme to control the corresponding smart home device.

[0020] According to another aspect of the present application, a control device for a smart home device is provided, the device comprising:

[0021] An acquisition unit, configured to acquire multimodal data, wherein the multimodal data represents data used to control the smart home device, and the multimodal data includes voice data, text data, and image data of a user;

[0022] A first processing unit, configured to preprocess each type of data in the multimodal data to obtain preprocessed data, and perform feature extraction on the preprocessed data to obtain feature data;

[0023] A second processing unit is used to train the capsule network model using the feature data to obtain a final capsule network model;

[0024] The third processing unit is used to apply the final capsule network model to control the smart home device accordingly.

[0025] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0026] According to another aspect of the present application, a control system for smart home devices is provided, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0027] By applying the technical solution of the present application, preprocessing is performed on each type of data in the multimodal data to obtain preprocessed data, and feature extraction is performed on the preprocessed data to obtain feature data, so that the feature-extracted data, i.e., the feature data, can better represent the user's true intention. The feature data is then used to train the capsule network model to obtain the final capsule network model, so that the present application can more accurately identify the user's true intention than the existing solution. Finally, the final capsule network model is used to control the smart home device accordingly, thereby solving the problem that the existing solution cannot enable the smart home to meet the user's true needs through commands in the voice mode alone. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings constituting part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 A schematic diagram of a process of controlling a smart home device provided according to an embodiment of the present application is shown;

[0030] Figure 2 A schematic diagram of a flow chart of another method for controlling a smart home device provided according to an embodiment of the present application is shown;

[0031] Figure 3 A structural block diagram of a control device for a smart home device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0036] Short-Time Fourier Transform (STFT) technology is a method that divides the signal into small blocks and performs Fourier transform on each small block. This method can analyze the signal in time and frequency, so as to obtain the spectrum information of the signal at different time points and frequencies.

[0037] Mel-frequency Cepstral Coefficients (MFCC) technology is a feature extraction method for audio signal processing. This method first maps the frequency value on the frequency axis to the Mel frequency axis, then performs Fourier transform on the signal on the Mel frequency axis, and finally extracts the cepstral coefficients of the transformed signal as features for applications such as speech recognition and audio processing. MFCC technology can effectively extract the features of audio signals, thereby realizing the analysis and processing of audio signals.

[0038] The bag-of-words model is a text feature extraction method used to convert text data into numerical data so that machine learning algorithms can process it. This technology converts words in the text into vector representations, regardless of the order and grammatical structure of the words, and only focuses on the frequency of word occurrence. Through the bag-of-words model technology, text data can be converted into a sparse matrix for training classifiers or regression models. This method is widely used in natural language processing tasks such as text classification, sentiment analysis, and information retrieval.

[0039] Framing: Splitting audio or video signals into several frames at fixed time intervals (frame length), commonly used in audio signal processing and video encoding and decoding.

[0040] Windowing: In signal processing, each frame of data is windowed, that is, multiplied by a window function, to reduce spectrum leakage and ringing in the frequency domain.

[0041] Word segmentation: In natural language processing, it refers to the process of dividing text into individual words or characters, which is often used in text analysis, information retrieval and other fields.

[0042] Cropping: Crop an image or video, that is, cut or delete part of the image or video to remove unnecessary content or adjust the image size.

[0043] Scaling: Scaling an image, video or audio signal, that is, adjusting its size or dimension, is commonly used in image processing, video editing and other fields.

[0044] One-hot vector is an encoding method for representing discrete features, in which each feature is represented as a vector with only one element being 1 and the rest being 0. This encoding method can avoid the impact of the size relationship between features on the model while also preserving the discreteness of the features. For example, if a feature has three values ​​{A, B, C}, it can be represented by a three-dimensional one-hot vector, which are [1,0,0], [0,1,0], and [0,0,1]. One-hot vectors are often used to process categorical features in machine learning.

[0045] As introduced in the background technology, in the field of smart home technology, users generally use voice control to control various smart homes. However, as the functions of smart home products continue to increase, voice control alone sometimes cannot allow smart homes to meet the real needs of users, that is, commands through voice alone cannot allow smart homes to meet the real needs of users. This often reduces the user experience when using smart homes, thereby affecting the reputation of the smart home. In order to solve the problem that the existing solution cannot allow smart homes to meet the real needs of users through commands through voice alone, the embodiments of the present application provide a control method, device, medium and system for smart home devices.

[0046] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0047] In this embodiment, a control method for a smart home device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Figure 1 It is a flowchart of a control method of a smart home device provided according to an embodiment of the present application.

[0049] like Figure 1 As shown, the method comprises the following steps:

[0050] Step S101, acquiring multimodal data, where the multimodal data represents data used to control the smart home device, and the multimodal data includes user voice data, text data, and image data;

[0051] Specifically, the image data may be an image of the user's lip shape, so that the user's intention can be identified based on the user's lip shape. By preparing multimodal data consisting of voice data, text data, and image data, the present application is more in line with the user's true intention than the existing solution;

[0052] Step S102, preprocessing each type of data in the multimodal data to obtain preprocessed data, and extracting features from the preprocessed data to obtain feature data;

[0053] Specifically, by processing each type of data differently, the data after feature extraction, that is, the feature data, can better represent the user's true intention.

[0054] In one embodiment of the present application, the preprocessed data includes first preprocessed data, second preprocessed data, and third preprocessed data. The preprocessing of each type of data in the multimodal data in step S102 to obtain the preprocessed data includes:

[0055] Performing word segmentation processing and normalization processing on the text data in the multimodal data in sequence to obtain the first preprocessed data;

[0056] The voice data in the multimodal data is sequentially subjected to frame division, windowing and normalization processing to obtain the second preprocessed data;

[0057] The above-mentioned image data in the above-mentioned multimodal data is cropped, scaled and normalized in sequence to obtain the above-mentioned third preprocessed data.

[0058] Specifically, in order to clean up invalid or abnormal data, the text data in the multimodal data is processed by word segmentation and normalization in turn, so that the text data is clearer and more understandable; the voice data in the multimodal data is processed by frame segmentation, windowing and normalization in turn to reduce spectrum leakage and ringing in the frequency domain; the image data in the multimodal data is cropped, scaled and normalized in turn, so that the image data can be more convenient for the subsequent capsule network to understand the correlation between data.

[0059] In one embodiment of the present application, the feature data includes first feature data, second feature data and third feature data, and feature extraction is performed on the preprocessed data to obtain feature data, including:

[0060] Using the bag-of-words model technology to extract features from the first preprocessed data to obtain the first feature data;

[0061] Using short-time Fourier transform technology and Mel-frequency cepstrum coefficient technology to extract features from the second preprocessed data to obtain the second feature data;

[0062] Convolutional neural network technology is used to extract features from the third preprocessed data to obtain the third feature data.

[0063] Specifically, the bag-of-words model technology is used to perform feature extraction on the above-mentioned first preprocessed data, that is, the text data is converted into numerical data so that the machine learning algorithm can process it, and the short-time Fourier transform technology and Mel-frequency cepstral coefficient technology are used to perform feature extraction on the above-mentioned second preprocessed data to effectively extract the features of the audio signal, thereby realizing the analysis and processing of the audio signal, and the convolutional neural network technology is used to perform feature extraction on the above-mentioned third preprocessed data (for example, using a pre-trained convolutional neural network model to extract high-level semantic features), so as to effectively extract features in the image.

[0064] Step S103, using the above feature data to train the capsule network model to obtain a final capsule network model;

[0065] The capsule network consists of capsule layers, each capsule represents the feature vector of a specific entity or concept.

[0066] Define capsule layers: Determine the number of capsule layers and the dimensions of each capsule in the capsule network. Capsules in lower layers capture local features, while capsules in higher layers represent more abstract, global features.

[0067] Initialize capsule: Assign an initial feature vector value to each capsule.

[0068] Capsule connection: Determine the connection method and weight between capsules.

[0069] In one embodiment of the present application, in the process of training the capsule network model using the above-mentioned feature data to obtain the final capsule network model, the above-mentioned method further includes:

[0070] The feature data is divided into a training set and a test set, and a dynamic routing algorithm is used to enable the capsule network model to learn the relationship and pattern between the different modal features in the training set;

[0071] Specifically, dynamic routing is the core mechanism of the capsule network, which is used to transfer and update information between capsule layers. The dynamic routing algorithm includes the following steps:

[0072] Step S201, for the connection between the lower layer capsule i and the higher layer capsule j, set the initial logarithmic prior probability b{ij} to 0;

[0073] Step S202, in each iteration, calculate the prediction vector p{ij}, which is the result of multiplying the feature vector of capsule i in the lower layer by the connection weight matrix;

[0074] Step S203, calculating the consistency value c{ij}, which can be obtained by normalizing all prediction vectors through a certain function (such as a softmax function);

[0075] Step S204, update the feature vector vj of the higher-level capsule j, which is the weighted sum of the prediction vectors p{ij} of all lower-level capsules multiplied by the consistency value c{ij};

[0076] Step S205, updating the logarithmic prior probability b{ij} according to the consistency value, for example, increasing b{ij} by a value proportional to the consistency value;

[0077] Step S206, repeat the above steps to a preset number of iterations, so that information is fully transferred and updated between capsule layers.

[0078] The weights of the above capsule network model are updated using the back-propagation algorithm to minimize the loss value of the loss function.

[0079] Specifically, in the process of updating the weights of the capsule network model using the back propagation algorithm to minimize the loss function, the method further includes:

[0080] The above loss function is constructed as:

[0081]

[0082] Among them, L is the loss value of the above loss function, K is the total number of dimensions, and y i is the true label of the i-th dimension, p i is the predicted probability distribution of the current capsule network model in the i-th dimension.

[0083] For example, there is a classification problem that is divided into K categories. For a given sample, the true label can be represented by a K-dimensional one-hot vector, in which only the element corresponding to the true category is 1 and the other elements are 0. The predicted capsule vector is passed through some activation function (such as softmax) to obtain the predicted probability distribution p, which is also a K-dimensional vector, indicating the probability of the sample belonging to each category.

[0084] For example, there are three categories, and the true label is category 2, that is, y = (0, 1, 0). For example, the predicted probability distribution is (0.3, 0.6, 0.1). Then the cross entropy loss is: L = -(0×log(0.3)+1×log(0.6)+0×log(0.1))=log(0.6)≈-0.511. This value indicates the degree of difference between the predicted result and the true label. The smaller the loss value, the closer the predicted result is to the true label.

[0085] In one embodiment of the present application, the capsule network model is trained using the above-mentioned feature data to obtain a final capsule network model, including:

[0086] Get the current precision and recall of the above capsule network model;

[0087] according to Determine the model performance coefficient;

[0088] The model performance coefficient represents the harmonic mean of the current precision and the current recall, F1 is the model performance coefficient, Precision is the current precision, and Recall is the current recall;

[0089] When the model performance coefficient is less than the coefficient threshold, determining the current capsule network model as the final capsule network model;

[0090] When the model performance coefficient is greater than or equal to the coefficient threshold, a prompt message is generated to prompt that at least the structure of the current capsule network model needs to be adjusted.

[0091] Specifically, TP (True Positives): True positives, the number of samples that are actually positive and predicted to be positive.

[0092] TN (True Negatives): True negative examples, the number of samples that are actually negative and predicted to be negative.

[0093] FP (False Positives): False positive examples, the number of samples that are actually negative but predicted to be positive.

[0094] FN (False Negatives): False negative examples, the number of samples that are actually positive but predicted to be negative.

[0095] Accuracy refers to the ratio of correctly classified samples to the total number of samples, and the calculation formula is:

[0096]

[0097] For example, if there are 100 test samples and 80 of them are correctly classified, then the accuracy is 80 / 100=0.8.

[0098] Recall rate is also called recall rate. For a certain category, it refers to the ratio of the number of samples correctly classified into that category to the total number of samples that actually belong to that category. The calculation formula for the recall rate of the positive class in the binary classification problem is:

[0099]

[0100] For example, in a binary classification problem, there are 50 positive samples, and the model correctly identifies 40 positive samples, then the recall rate is 40 / 50=0.8.

[0101] The F1 value is the harmonic mean of precision and recall, which takes both precision and recall into consideration.

[0102] The calculation formula for the positive class in the binary classification problem is:

[0103]

[0104] For example, if the model predicts 60 samples as positive, and 40 of them are indeed positive, then the precision is 40 / 60≈0.67. If the recall is 0.8, then F1 is 2×(0.67×0.8) / (0.67+0.8)≈0.73.

[0105] In addition, for a multi-intention detection task of an intelligent voice assistant, for example, a set of feature vectors is obtained after feature extraction of voice data, and corresponding feature vectors are obtained after feature extraction of image data through convolutional neural network technology. These feature vectors enter the capsule network as input. In the capsule network, through the dynamic routing algorithm, capsules at the lower level (such as capsules representing certain local features of voice and certain local area features of images) continuously interact and transfer information with capsules at the higher level. After multiple iterations, high-level capsules can capture the comprehensive relationship between voice and image features, thereby representing more abstract intent features. Finally, based on the feature vectors of high-level capsules, the intent corresponding to the input voice and image combination can be determined, such as querying the weather, playing music, etc. By continuously training and optimizing the capsule network, the accuracy of intent detection for various multimodal inputs can be improved.

[0106] Step S104: Apply the final capsule network model to control the smart home devices accordingly.

[0107] In the above steps, preprocessing is performed on each type of data in the above multimodal data to obtain preprocessed data, and feature extraction is performed on the above preprocessed data to obtain feature data, so that the data after feature extraction, that is, the feature data, can better represent the user's true intention. Then, the capsule network model is trained using the above feature data to obtain the final capsule network model, so that the present application can more accurately identify the user's true intention than the existing solution. Finally, the above final capsule network model is used to control the above smart home devices accordingly, thereby solving the problem that the existing solution cannot enable the smart home to meet the user's true needs through the instructions of the voice mode alone.

[0108] Specifically, the current multimodal data is obtained; the final capsule network model is used to process the current multimodal data to obtain a control scheme; and the control scheme is used to control the corresponding smart home device. Thus, compared with the existing scheme, the present application considers more image data and improves the accuracy of the finally generated control scheme.

[0109] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the control method of the smart home device of the present application will be described in detail below in combination with specific embodiments.

[0110] This embodiment relates to a specific control method for smart home devices, such as Figure 2 As shown, the following steps are included:

[0111] Acquire multimodal data, where the multimodal data represents data used to control smart home devices, and the multimodal data includes user voice data, text data, and image data;

[0112] The text data in the multimodal data is sequentially subjected to word segmentation and normalization processing to obtain first preprocessed data; the voice data in the multimodal data is sequentially subjected to frame segmentation, windowing and normalization processing to obtain second preprocessed data; the image data in the multimodal data is sequentially subjected to cropping, scaling and normalization processing to obtain third preprocessed data, the preprocessed data including first preprocessed data, second preprocessed data and third preprocessed data,

[0113] The first preprocessed data is subjected to feature extraction using a bag-of-words model technology to obtain first feature data; the second preprocessed data is subjected to feature extraction using a short-time Fourier transform technology and a Mel-frequency cepstral coefficient technology to obtain second feature data; the third preprocessed data is subjected to feature extraction using a convolutional neural network technology to obtain third feature data, wherein the feature data includes the first feature data, the second feature data and the third feature data;

[0114] The feature data is divided into a training set and a test set, and a dynamic routing algorithm is used to enable the capsule network model to learn the relationship and pattern between different modal features in the training set; the weight of the capsule network model is updated using the back-propagation algorithm to minimize the loss value of the loss function;

[0115] Get the current precision and recall of the current capsule network model;

[0116] according to Determine the model performance coefficient;

[0117] Among them, the model performance coefficient represents the harmonic mean of the current precision and the current recall rate, F1 is the model performance coefficient, Precision is the current precision rate, and Recall is the current recall rate;

[0118] When the model performance coefficient is less than the coefficient threshold, the current capsule network model is determined to be the final capsule network model; when the model performance coefficient is greater than or equal to the coefficient threshold, a prompt message is generated to prompt that at least the structure of the current capsule network model needs to be adjusted;

[0119] Specifically, in the process of updating the weights of the capsule network model using the back propagation algorithm to minimize the loss function, the method further includes:

[0120] The loss function is constructed as:

[0121]

[0122] Among them, L is the loss value of the loss function, K is the total number of dimensions, and y i is the true label of the i-th dimension, p i is the predicted probability distribution of the current capsule network model in the i-th dimension.

[0123] Acquire current multimodal data; use the final capsule network model to process the current multimodal data to obtain a control scheme; and use the control scheme to control corresponding smart home devices.

[0124] By preprocessing each type of data in the multimodal data to obtain preprocessed data, and extracting features from the preprocessed data to obtain feature data, the feature-extracted data, i.e., the feature data, can better represent the user's true intentions. The feature data is then used to train the capsule network model to obtain the final capsule network model, so that the present application can more accurately identify the user's true intentions than the existing solutions. Finally, the final capsule network model is used to control the smart home devices accordingly, thereby solving the problem that the existing solutions cannot enable the smart home to meet the user's true needs through commands in the voice mode alone.

[0125] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0126] The embodiment of the present application also provides a control device for a smart home device. It should be noted that the control device for the smart home device in the embodiment of the present application can be used to execute the control method for the smart home device provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0127] The following is an introduction to the control device of the smart home device provided in the embodiment of the present application.

[0128] Figure 3 1 is a block diagram of a control device for a smart home device provided according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0129] An acquisition unit 31 is used to acquire multimodal data, where the multimodal data represents data used to control the smart home device, and the multimodal data includes user voice data, text data, and image data;

[0130] The first processing unit 32 is used to preprocess each type of data in the multimodal data to obtain preprocessed data, and perform feature extraction on the preprocessed data to obtain feature data;

[0131] The second processing unit 33 is used to train the capsule network model using the above-mentioned feature data to obtain a final capsule network model;

[0132] The third processing unit 34 is used to apply the final capsule network model to control the smart home device accordingly.

[0133] In the above-mentioned device, preprocessing data is obtained by preprocessing each type of data in the above-mentioned multimodal data, and feature extraction is performed on the above-mentioned preprocessed data to obtain feature data, so that the data after feature extraction, that is, the feature data, can better represent the user's true intention. Then, the capsule network model is trained using the above-mentioned feature data to obtain the final capsule network model, so that the present application can more accurately identify the user's true intention than the existing solution, and finally the above-mentioned final capsule network model is used to control the above-mentioned smart home devices accordingly, thereby solving the problem that the existing solution cannot make the smart home meet the user's true needs through the instructions of the voice mode alone.

[0134] In one embodiment of the present application, the preprocessed data includes first preprocessed data, second preprocessed data and third preprocessed data, the first processing unit includes a first processing module, a second processing module and a third processing module,

[0135] The first processing module is used to perform word segmentation processing and normalization processing on the text data in the multimodal data in sequence to obtain the first preprocessed data;

[0136] The second processing module is used to perform frame division, windowing and normalization processing on the above-mentioned voice data in the above-mentioned multimodal data in sequence to obtain the above-mentioned second pre-processed data;

[0137] The third processing module is used to perform cropping, scaling and normalization processing on the above-mentioned image data in the above-mentioned multimodal data in sequence to obtain the above-mentioned third pre-processed data.

[0138] In one embodiment of the present application, the feature data includes first feature data, second feature data and third feature data, the first processing unit includes a fourth processing module, a fifth processing module and a sixth processing module,

[0139] The fourth processing module is used to extract features from the first preprocessed data using a bag-of-words model to obtain the first feature data.

[0140] The fifth processing module is used to extract features from the second pre-processed data using short-time Fourier transform technology and Mel frequency cepstral coefficient technology to obtain the second feature data;

[0141] The sixth processing module is used to use convolutional neural network technology to perform feature extraction on the above-mentioned third preprocessed data to obtain the above-mentioned third feature data.

[0142] In one embodiment of the present application, the seventh processing module and the eighth processing module of the second processing unit,

[0143] The seventh processing module is used to divide the feature data into a training set and a test set in the process of training the capsule network model using the feature data to obtain the final capsule network model, and adopt a dynamic routing algorithm so that the capsule network model learns the relationship and pattern between the different modal features in the training set;

[0144] The eighth processing module is used to update the weights of the capsule network model using a back propagation algorithm to minimize the loss value of the loss function.

[0145] In one embodiment of the present application, the eighth processing module includes a construction submodule,

[0146] The construction submodule is used to construct the above loss function in the process of updating the weights of the above capsule network model using the back propagation algorithm to minimize the loss function:

[0147]

[0148] Among them, L is the loss value of the above loss function, K is the total number of dimensions, and y i is the true label of the i-th dimension, p i is the predicted probability distribution of the current capsule network model in the i-th dimension.

[0149] In one embodiment of the present application, the second processing unit includes a first acquisition module, a first determination module, a second determination module and a third determination module.

[0150] The first acquisition module is used to obtain the current precision and current recall of the current capsule network model;

[0151] The first determining module is used to Determine the model performance coefficient;

[0152] The model performance coefficient represents the harmonic mean of the current precision and the current recall, F1 is the model performance coefficient, Precision is the current precision, and Recall is the current recall;

[0153] The second determination module is used to determine that the current capsule network model is the final capsule network model when the model performance coefficient is less than the coefficient threshold;

[0154] The third determination module is used to generate prompt information when the model performance coefficient is greater than or equal to the coefficient threshold, so as to prompt that at least the structure of the current capsule network model needs to be adjusted.

[0155] In one embodiment of the present application, the third processing unit includes a second acquisition module, a ninth processing module and a tenth processing module.

[0156] The second acquisition module is used to acquire current multimodal data;

[0157] The ninth processing module is used to process the current multimodal data using the final capsule network model to obtain a control solution;

[0158] The tenth processing module is used to control the corresponding smart home device using the above control scheme.

[0159] The control device of the above-mentioned smart home device includes a processor and a memory, and the above-mentioned acquisition unit, the first processing unit, the second processing unit and the third processing unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0160] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the existing solution cannot meet the real needs of users through voice commands alone.

[0161] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0162] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the control method of the smart home device.

[0163] An embodiment of the present invention provides a processor, which is used to run a program, wherein the control method of the smart home device is executed when the program is run.

[0164] The embodiment of the present invention provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented: obtaining multimodal data, wherein the multimodal data represents data used to control the smart home device, and the multimodal data includes voice data, text data, and image data of the user; preprocessing each type of data in the multimodal data to obtain preprocessed data, and extracting features from the preprocessed data to obtain feature data; training a capsule network model using the feature data to obtain a final capsule network model; and applying the final capsule network model to control the smart home device accordingly. The device in this article may be a server, a PC, a PAD, a mobile phone, etc.

[0165] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes at least the following method steps: obtaining multimodal data, the multimodal data representing data used to control the smart home device, the multimodal data including user voice data, text data and image data; preprocessing each type of data in the multimodal data to obtain preprocessed data, and extracting features from the preprocessed data to obtain feature data; training a capsule network model using the feature data to obtain a final capsule network model; and applying the final capsule network model to control the smart home device accordingly.

[0166] The present application also provides a control system for smart home devices, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the above methods. Preprocessing each type of data in the multimodal data to obtain preprocessed data, and extracting features from the preprocessed data to obtain feature data, so that the feature-extracted data, that is, the feature data, can better represent the user's true intentions, and then the feature data is used to train the capsule network model to obtain the final capsule network model, so that the present application can more accurately identify the user's true intentions than the existing solutions, and finally the final capsule network model is used to control the smart home devices accordingly, thereby solving the problem that the existing solutions cannot make the smart home meet the user's true needs through voice commands alone.

[0167] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0168] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0169] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0172] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0173] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0174] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0175] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0176] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0177] 1) The control method of the smart home device of the present application preprocesses each type of data in the above-mentioned multimodal data to obtain preprocessed data, and extracts features from the above-mentioned preprocessed data to obtain feature data, so that the data after feature extraction, that is, the feature data, can better represent the user's true intention. Then, the capsule network model is trained using the above-mentioned feature data to obtain the final capsule network model, so that the present application can more accurately identify the user's true intention than the existing solution, and finally the above-mentioned final capsule network model is used to control the above-mentioned smart home devices accordingly, thereby solving the problem that the existing solution cannot make the smart home meet the real needs of the user through the instructions of the voice mode alone.

[0178] 2) The control device of the smart home device of the present application preprocesses each type of data in the above-mentioned multimodal data to obtain preprocessed data, and extracts features from the above-mentioned preprocessed data to obtain feature data, so that the data after feature extraction, that is, the feature data, can better represent the user's true intention. Then, the capsule network model is trained using the above-mentioned feature data to obtain the final capsule network model, so that the present application can more accurately identify the user's true intention than the existing solution, and finally the above-mentioned final capsule network model is used to control the above-mentioned smart home devices accordingly, thereby solving the problem that the existing solution cannot make the smart home meet the real needs of the user through the instructions of the voice mode alone.

[0179] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A control method for a smart home device, characterized in that: include: Acquire multimodal data, where the multimodal data represents data used to control the smart home device, and the multimodal data includes voice data, text data, and image data of a user; Preprocessing each type of data in the multimodal data to obtain preprocessed data, and extracting features from the preprocessed data to obtain feature data; Using the feature data to train the capsule network model to obtain a final capsule network model; The final capsule network model is applied to control the smart home device accordingly.

2. The method according to claim 1, characterized in that The preprocessed data includes first preprocessed data, second preprocessed data and third preprocessed data, and each type of data in the multimodal data is preprocessed to obtain preprocessed data, including: Performing word segmentation processing and normalization processing on the text data in the multimodal data in sequence to obtain the first preprocessed data; performing frame division, windowing and normalization processing on the speech data in the multimodal data in sequence to obtain the second preprocessed data; The image data in the multimodal data is cropped, scaled, and normalized in sequence to obtain the third preprocessed data.

3. The method according to claim 2, characterized in that The feature data includes first feature data, second feature data and third feature data, and feature extraction is performed on the preprocessed data to obtain feature data, including: Using a bag-of-words model technique to perform feature extraction on the first preprocessed data to obtain the first feature data; Using short-time Fourier transform technology and Mel-frequency cepstral coefficient technology to extract features from the second preprocessed data to obtain the second feature data; Convolutional neural network technology is used to perform feature extraction on the third preprocessed data to obtain the third feature data.

4. The method according to claim 1, characterized in that: In the process of training the capsule network model using the feature data to obtain the final capsule network model, the method further includes: Dividing the feature data into a training set and a test set, and using a dynamic routing algorithm so that the capsule network model learns the relationship and pattern between different modal features in the training set; The weights of the capsule network model are updated using a back-propagation algorithm to minimize the loss value of the loss function.

5. The method according to claim 4, characterized in that In the process of updating the weights of the capsule network model using a back propagation algorithm to minimize the loss function, the method further includes: The loss function is constructed as: Among them, L is the loss value of the loss function, K is the total number of dimensions, and y i is the true label of the i-th dimension, p i is the predicted probability distribution of the current capsule network model in the i-th dimension.

6. The method according to claim 1, characterized in that The capsule network model is trained using the feature data to obtain a final capsule network model, including: Obtain the current precision and current recall of the current capsule network model; according to Determine the model performance coefficient; The model performance coefficient represents the harmonic mean of the current precision and the current recall, F1 is the model performance coefficient, Precision is the current precision, and Recall is the current recall; When the model performance coefficient is less than the coefficient threshold, determining the current capsule network model as the final capsule network model; When the model performance coefficient is greater than or equal to the coefficient threshold, prompt information is generated to prompt that at least the structure of the current capsule network model needs to be adjusted.

7. The method according to any one of claims 1 to 6, characterized in that Applying the final capsule network model to control the smart home device accordingly includes: Get current multimodal data; Using the final capsule network model to process the current multimodal data to obtain a control solution; The control scheme is adopted to control the corresponding smart home device.

8. A control device for a smart home device, characterized in that: include: An acquisition unit, configured to acquire multimodal data, wherein the multimodal data represents data used to control the smart home device, and the multimodal data includes voice data, text data, and image data of a user; A first processing unit, configured to preprocess each type of data in the multimodal data to obtain preprocessed data, and perform feature extraction on the preprocessed data to obtain feature data; A second processing unit is used to train the capsule network model using the feature data to obtain a final capsule network model; The third processing unit is used to apply the final capsule network model to control the smart home device accordingly.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A control system for smart home devices, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images