Intelligent equipment control method, device, medium and system

Abstract

The invention provides a control method, device and system for intelligent equipment and a medium. According to the method, voice data, image data and data of the environment where the user is located are considered, so that the accuracy of understanding the intention of the user is improved compared with an existing scheme, feature extraction is carried out on various types of data in the user data to obtain a comprehensive feature vector, and the user experience is improved. And processing the comprehensive feature vector by at least adopting a convolutional neural network technology and a long short-term memory network technology to obtain a final control strategy, so that compared with the existing scheme, the real intention of the user can be better understood from multiple levels, thereby improving the accuracy of controlling the intelligent equipment and improving the user experience. Therefore, the problem that the existing scheme controls the intelligent device only according to the voice of the user, so that the control result of the existing scheme on the intelligent device cannot meet the real demand of the user, and the accuracy of the existing scheme on the control of the intelligent device is low is solved.

Description

Technical Field

[0001] The present application relates to a smart device, and more specifically, to a control method, device, medium and system for a smart device. Background Art

[0002] With the popularization of applications such as smart homes and smart offices, multi-user and multi-device scenarios are becoming increasingly complex. Most existing intelligent control systems rely solely on a single feature, such as voice or image, and it is difficult to respond accurately and quickly in complex scenarios. The comprehensive use of multiple information such as user voice commands, facial expressions, behavioral habits, etc. has not been effectively utilized by existing solutions, and most existing solutions only control smart devices based on the user's voice, resulting in the existing solutions The results of controlling smart devices cannot meet the real needs of users, which in turn leads to low accuracy in controlling smart devices. At present, there is a lack of an intelligent control method that can effectively integrate multiple feature information. Summary of the invention

[0003] The main purpose of the present application is to provide a method, device, medium and system for controlling a smart device, so as to at least solve the problem that the existing solution controls the smart device only according to the user's voice, resulting in the result of the existing solution controlling the smart device cannot meet the real needs of the user, and further resulting in the low accuracy of the existing solution controlling the smart device.

[0004] In order to achieve the above object, according to one aspect of the present application, a control method for a smart device is provided, the method comprising:

[0005] Acquire user data, the user data at least including user voice data, user image data, and user environment data, the user environment data including temperature, humidity, and light intensity of the user environment, and the user image data including the user's facial image and / or the user's lip shape change image;

[0006] Extracting features of various types of data in the user data to obtain a comprehensive feature vector;

[0007] At least using convolutional neural network technology and long short-term memory network technology to process the comprehensive feature vector to obtain a final control strategy;

[0008] The final control strategy is used to control the corresponding intelligent device to perform corresponding actions.

[0009] Optionally, feature extraction is performed on each type of data in the user data to obtain a comprehensive feature vector, including:

[0010] After the voice data of the user in the user data is converted into text, the instructions and the emotion information of the user in the obtained text data are extracted using natural language preprocessing technology to obtain a text feature vector;

[0011] Using convolutional neural network technology to extract facial features and / or lip shape change features of the user in the image data of the user in the user data to obtain an image feature vector;

[0012] Using a neural network model technology to extract features of the user's environment data from the user data to obtain an environment feature vector;

[0013] The text feature vector, the image feature vector and the environment feature vector are fused to obtain the comprehensive feature vector.

[0014] Optionally, at least a convolutional neural network technology and a long short-term memory network technology are used to process the comprehensive feature vector to obtain a final control strategy, including:

[0015] A hybrid model is used to extract deep features from the comprehensive feature vector to obtain a final comprehensive feature vector, wherein the hybrid model is a model combining a convolutional neural network and a long short-term memory network, and the deep features are features of a preset level;

[0016] The final control strategy is generated according to the final comprehensive feature vector.

[0017] Optionally, at least a convolutional neural network technology and a long short-term memory network technology are used to process the comprehensive feature vector to obtain a final control strategy, including:

[0018] A hybrid model is used to extract deep features from the comprehensive feature vector to obtain an intermediate comprehensive feature vector, wherein the hybrid model is a model combining a convolutional neural network and a long short-term memory network;

[0019] Using a feature selection algorithm to remove redundant features in the intermediate comprehensive feature vector to obtain a final comprehensive feature vector, the feature selection algorithm including a random forest algorithm and a decision tree;

[0020] The final control strategy is generated according to the final comprehensive feature vector.

[0021] Optionally, generating the final control strategy according to the final comprehensive feature vector includes:

[0022] Using the hybrid model to classify and identify the final comprehensive feature vector to obtain the specific intention of the user;

[0023] The final control strategy is generated according to the specific intention of the user.

[0024] Optionally, the user data further includes behavior data of the user, and the behavior data of the user includes the number of steps, heart rate and activity time of the user. In the process of extracting features of various types of data in the user data to obtain a comprehensive feature vector, the method further includes:

[0025] The machine learning model technology is used to extract the user's behavior pattern characteristics from the behavior data of the user in the user data to obtain the behavior vector characteristics.

[0026] Optionally, the user data further includes device information of the user, and the device information of the user includes chat records and calendar events of the user. In the process of extracting features of various types of data in the user data to obtain a comprehensive feature vector, the method further includes:

[0027] The machine learning model technology is used to extract the user's device behavior characteristics from the user's device information in the user data to obtain device vector characteristics.

[0028] According to another aspect of the present application, a control device for a smart device is provided, the device comprising:

[0029] an acquisition unit, configured to acquire user data, wherein the user data at least includes user voice data, user image data, and user environment data, wherein the user environment data includes temperature, humidity, and light intensity of the user environment, and the user image data includes a facial image of the user and / or an image of the user's lip shape changes;

[0030] A first processing unit, configured to extract features from each type of data in the user data to obtain a comprehensive feature vector;

[0031] A second processing unit is used to process the comprehensive feature vector using at least a convolutional neural network technology and a long short-term memory network technology to obtain a final control strategy;

[0032] The third processing unit is used to control the corresponding intelligent device to perform corresponding actions by adopting the final control strategy.

[0033] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0034] According to another aspect of the present application, a control system for an intelligent device is provided, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0035] By applying the technical solution of the present application, voice data, image data, and user environment data are taken into consideration, thereby improving the accuracy of understanding the user's intention compared to the existing solutions. Feature extraction is performed on various types of data in the user data to obtain a comprehensive feature vector. The comprehensive feature vector is processed by at least convolutional neural network technology and long short-term memory network technology to obtain a final control strategy. Compared with the existing solutions, the user's true intention can be understood from multiple levels, thereby improving the accuracy of controlling smart devices, thereby solving the problem that the existing solutions control smart devices only according to the user's voice, resulting in the existing solutions The results of controlling smart devices cannot meet the real needs of the user, which in turn leads to the low accuracy of controlling smart devices by the existing solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 A schematic diagram of a flow chart of a control method of a smart device provided according to an embodiment of the present application is shown;

[0038] Figure 2 A schematic diagram showing a flow chart of another method for controlling a smart device provided according to an embodiment of the present application is shown;

[0039] Figure 3 A structural block diagram of a control device for an intelligent device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0044] The model that combines Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) is usually called a hybrid model or CNN-LSTM model. This model combines the powerful processing ability of CNN for spatial information and the modeling ability of LSTM for time series information, and can be used to process data that contains both spatial and temporal information, such as video, audio, etc. This model has achieved good performance in many fields, such as video analysis, natural language processing, etc.

[0045] A decision tree is a tree-structured model that classifies or predicts data through a series of conditional judgments. It makes predictions by dividing the data into different regions and applying different rules in each region. Decision trees are easy to understand and explain, but they are prone to overfitting.

[0046] The random forest algorithm is an ensemble learning method that makes predictions based on an ensemble of multiple decision trees. When building each decision tree, the random forest algorithm randomly samples the training data and randomly selects features to split at each node. By combining the prediction results of multiple decision trees, random forest can reduce the risk of overfitting and improve the generalization ability of the model.

[0047] Machine learning model technology is an artificial intelligence technology that allows computers to automatically discover patterns and laws in data through learning and training of large amounts of data, and make predictions and decisions based on these laws. Machine learning model technology includes different types of learning methods such as supervised learning, unsupervised learning, and reinforcement learning. Commonly used models include neural networks, decision trees, support vector machines, and logistic regression. These models can be used in various fields, such as natural language processing, image recognition, and financial forecasting, and have achieved remarkable results in many practical applications.

[0048] As introduced in the background technology, with the popularization of applications such as smart home and smart office, the scenarios of multiple users and multiple devices are becoming increasingly complex. Most of the existing intelligent control systems rely solely on a certain feature, such as voice or image, and it is difficult to achieve accurate and rapid response in complex scenarios. The comprehensive use of multiple information such as user voice commands, facial expressions, behavioral habits, etc. has not been effectively utilized by existing solutions, and most of the existing solutions only control smart devices based on the user's voice, which results in the existing solutions controlling smart devices unable to meet the user's real needs, and further leads to the low accuracy of the existing solutions controlling smart devices. At present, there is a lack of an intelligent control method that can effectively integrate multiple feature information. In order to solve the problem that the existing solutions only control smart devices based on the user's voice, which results in the existing solutions controlling smart devices unable to meet the user's real needs, and further leads to the low accuracy of the existing solutions controlling smart devices, the embodiments of the present application provide a control method, device, medium and system for smart devices.

[0049] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0050] In this embodiment, a control method for an intelligent device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] Figure 1 FIG. 1 is a flow chart of a control method of a smart device provided according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0052] Step S101, obtaining user data, the user data at least including user voice data, user image data, and user environment data, the user environment data including temperature, humidity, and light intensity of the user environment, and the user image data including the user's facial image and / or the user's lip shape change image;

[0053] Specifically, the user data may also include the device information of the user, the device information of the user may include the chat history and calendar events of the user, the user data may also include the behavior data of the user, the behavior data of the user may include the number of steps, heart rate and activity time of the user;

[0054] In actual operation, voice data of multiple users are collected through a microphone array, and the voice data is processed by noise filtering and voice enhancement as voice data; image data is collected by acquiring the user's facial image and lip shape changes through a high-definition camera as image data; text information is extracted from the user's device (such as mobile phone, computer), including chat records, calendar events, etc., to collect text data as device information; user behavior data such as step count, heart rate, activity time, etc. are obtained through devices such as smart bracelets and smart watches to collect behavior habit data as behavior data; environmental sensors are used to acquire information such as temperature, humidity, light, etc. of the surrounding environment to collect environmental data as environmental data.

[0055] By taking voice data, image data, and user environment data into consideration, the accuracy of the understanding of user intent in this application is improved compared to existing solutions.

[0056] Step S102, extracting features from each type of data in the above user data to obtain a comprehensive feature vector;

[0057] Step S102, namely extracting features from each type of data in the above user data to obtain a comprehensive feature vector, including:

[0058] After the voice data of the user in the user data is converted into text, the instructions and the emotion information of the user in the obtained text data are extracted using natural language preprocessing technology to obtain a text feature vector;

[0059] Among them, speech recognition technology is usually used to convert speech to text. This technology uses machine learning and natural language processing algorithms to convert speech input into text output. Common speech recognition technologies include rule-based methods, statistical models, and deep learning models. These technologies can achieve speech-to-text conversion through specialized speech recognition software or APIs.

[0060] Using convolutional neural network technology to extract facial features and / or lip shape change features of the user in the image data of the user in the user data to obtain an image feature vector;

[0061] Using a neural network model technology to extract features of the user's environment data from the user data to obtain an environment feature vector;

[0062] The text feature vector, the image feature vector and the environment feature vector are fused to obtain the comprehensive feature vector.

[0063] Specifically, speech recognition technology is used to convert voice data into text, and key instructions and emotional information in the voice are extracted; convolutional neural networks (CNN) are used for image processing to extract the user's facial features and lip shape changes; natural language processing (NLP) technology is used to analyze text data and extract instructions and emotional information; machine learning models are used to analyze user behavior data and extract the user's typical behavior patterns; and data from environmental sensors are analyzed.

[0064] Command: When a user enters "Wake up person C at A tomorrow morning" in the chat history of the smart home assistant, the system will recognize "Wake up person C" as a key command and extract "A tomorrow morning" as the time information from the text. This command can be used to set a smart alarm clock, allowing the system to automatically wake up the user at a specified time.

[0065] Emotional information: When a user tells the smart assistant during a chat, "I'm very tired today and don't want to cook." Through emotional analysis, the system can recognize that "very tired" represents fatigue, infer the user's emotions based on the fatigue state, and infer that the user does not want to do housework at this time. This may trigger the system to suggest that the user order takeout or automatically play soothing music for the user to help relax.

[0066] In one embodiment of the present application, the user data further includes the user's behavior data, and the user's behavior data includes the user's step count, heart rate, and activity time. In the process of extracting features from each type of data in the user data to obtain a comprehensive feature vector, the method further includes:

[0067] The machine learning model technology is used to extract the user's behavior pattern characteristics from the above-mentioned behavior data of the above-mentioned user in the above-mentioned user data to obtain the behavior vector characteristics.

[0068] Specifically, by adopting machine learning model technology to extract the user's behavior pattern characteristics from the above-mentioned behavior data of the above-mentioned user in the above-mentioned user data, it is possible to mine information that affects the user's intention to control the smart device from the behavior data.

[0069] In one embodiment of the present application, the user data further includes device information of the user, and the device information of the user includes chat records and calendar events of the user. In the process of extracting features of various types of data in the user data to obtain a comprehensive feature vector, the method further includes:

[0070] The machine learning model technology is used to extract the user's device behavior characteristics from the above-mentioned device information of the above-mentioned user in the above-mentioned user data to obtain the device vector characteristics.

[0071] Specifically, by adopting machine learning model technology to extract the user's device behavior characteristics from the above-mentioned device information of the above-mentioned user in the above-mentioned user data, it is possible to mine information that affects the user's intention to control the smart device from the device information.

[0072] User behavior patterns composed of user behavior characteristics include:

[0073] Daily routine: If the system detects that the user gets up at A in the morning and goes to bed at around B in the evening, this can be used as the user's daily routine. Based on this routine, the system can open the curtains and adjust the room temperature before the user gets up, or automatically dim the lights and switch to "quiet mode" before the user falls asleep.

[0074] Eating time: For example, if the user is often in the kitchen at C in the afternoon and D in the evening, the system can infer that these time periods are meal times. The system can automatically adjust the lighting brightness in the kitchen during these time periods and play light music during dinner to create a comfortable dining environment.

[0075] Exercise frequency: If the user walks or runs regularly between E and F in the afternoon every day (measured by the number of steps and heart rate changes monitored by the smart bracelet), the system can identify this period as the user's exercise time. The system can provide exercise suggestions during this period, or push weather forecast information suitable for exercise.

[0076] Health status monitoring: By monitoring the user's heart rate, sleep quality, activity level and other data over a preset period of time (e.g., within a week), the system can discover the user's health status pattern. For example, if the user's heart rate is high and the sleep quality is poor for a week in a row, the system can push relaxation tips or remind the user to adjust his or her work and rest schedule.

[0077] The comprehensive feature vector can be obtained by fusing various types of vector features.

[0078] Step S103, at least using convolutional neural network technology and long short-term memory network technology to process the above comprehensive feature vector to obtain a final control strategy;

[0079] There are two specific implementations of step S103:

[0080] The first specific implementation method:

[0081] A hybrid model is used to extract deep features from the comprehensive feature vector to obtain a final comprehensive feature vector, wherein the hybrid model is a model combining a convolutional neural network and a long short-term memory network, and the deep features are features of a preset level;

[0082] The above-mentioned final control strategy is generated according to the final comprehensive feature vector.

[0083] Specifically, the preset level of features allows users to make corresponding settings, and only uses a model combining convolutional neural networks and long short-term memory networks to extract deep-level features from the above-mentioned comprehensive feature vectors, which can more accurately explore the user's true intentions than existing solutions.

[0084] The second specific implementation method:

[0085] A hybrid model is used to extract deep features from the comprehensive feature vector to obtain an intermediate comprehensive feature vector, wherein the hybrid model is a model combining a convolutional neural network and a long short-term memory network;

[0086] Using a feature selection algorithm to remove redundant features in the intermediate comprehensive feature vector to obtain a final comprehensive feature vector, the feature selection algorithm includes a random forest algorithm and a decision tree;

[0087] The final control strategy is generated according to the final comprehensive feature vector.

[0088] Specifically, a feature selection algorithm is used to remove redundant features in the above intermediate comprehensive feature vector: a random forest algorithm or a decision tree is used to calculate the importance score of each feature and evaluate the contribution of each feature to the classification of user intent. For example, if the contribution of voice, emotion, heart rate changes, and device usage frequency to predicting whether the user wants to relax (such as playing music or dimming the lights) exceeds a set threshold, the importance scores of these features will be higher. Then, dimensionality reduction techniques such as principal component analysis or linear discriminant analysis are used to reduce the dimension of the features, retain the features that are most useful for intent judgment, and remove redundant or irrelevant features. The second specific implementation method is more accurate than the first specific implementation method.

[0089] Impact of user health status on intelligent control:

[0090] Preliminary feature set: includes the user’s heart rate, number of steps, sleep quality, temperature preference, device usage frequency, etc.

[0091] Feature selection process: The importance of these features is evaluated using the random forest algorithm, and it is found that heart rate and sleep quality contribute more to the judgment of the user's health status, while the number of steps has a smaller impact. The system will prioritize the use of heart rate and sleep quality data to infer whether the user needs to adjust the air conditioning temperature or play soothing music.

[0092] Results: The step count feature was removed, while the heart rate and sleep quality features were retained, which improved the prediction accuracy of the model and optimized the control strategy of the system.

[0093] In one embodiment of the present application, generating the final control strategy according to the final comprehensive feature vector includes:

[0094] The above hybrid model is used to classify and identify the above final comprehensive feature vector to obtain the user's specific intention;

[0095] The final control strategy is generated according to the specific intention of the user.

[0096] A conventional model combining a convolutional neural network and a long short-term memory network can be used to generate the above-mentioned final control strategy according to the above-mentioned specific intention of the above-mentioned user.

[0097] Step S104: Use the above final control strategy to control the corresponding smart device to perform corresponding actions.

[0098] The final control strategy, as an instruction, can be transmitted to various smart devices through the IoT platform to perform corresponding operations.

[0099] In the above steps, by taking voice data, image data, and user environment data into consideration, the accuracy of understanding user intentions in the present application is improved compared to the existing solutions. Feature extraction is performed on each type of data in the above user data to obtain a comprehensive feature vector. The above comprehensive feature vector is processed by at least convolutional neural network technology and long short-term memory network technology to obtain the final control strategy. Compared with the existing solutions, the user's true intentions can be understood from multiple levels, thereby improving the accuracy of controlling smart devices, thereby solving the problem that the existing solutions control smart devices only according to the user's voice, resulting in the existing solutions The results of controlling smart devices cannot meet the real needs of users, which in turn leads to the low accuracy of the existing solutions in controlling smart devices.

[0100] Smart devices can be smart home devices such as air conditioners, refrigerators, and televisions.

[0101] In addition, the control strategy can be dynamically adjusted according to the user's real-time feedback and environmental changes to optimize the user experience: continuously monitor user behavior and feedback and collect real-time data. Based on real-time data and user feedback information, the control strategy is dynamically adjusted using reinforcement learning algorithms to optimize the system's responsiveness and user experience. The system continuously learns the user's behavior habits and preferences, makes personalized adjustments, and improves user satisfaction.

[0102] For example, in a smart home scenario, users control smart devices in their home through voice, gestures, facial expressions, and other methods. The system collects user voice and image data through microphones and cameras, and combines user behavior habit data (such as the activity time every morning) to generate a comprehensive feature vector. According to the user's intention, the system automatically adjusts the light brightness, temperature settings, and music playback to achieve personalized smart home control. For example, when the user returns home, the system recognizes the user's facial features and voice commands, and can automatically turn on the lights in the home, adjust the air conditioning temperature, and play the user's favorite music.

[0103] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the control method of the smart device of the present application will be described in detail below in combination with specific embodiments.

[0104] This embodiment relates to a specific control method of an intelligent device, such as Figure 2 As shown, the following steps are included:

[0105] Acquire user data, where the user data includes the user's voice data, the user's image data, the user's environment data, the user's behavior data, and the user's device information. The user's device information includes the user's chat history and calendar events. The user's behavior data includes the user's steps, heart rate, and activity time. The user's environment data includes the temperature, humidity, and light intensity of the user's environment. The user's image data includes the user's facial image and / or the user's lip shape change image.

[0106] After the user's voice data in the user data is converted into text, the natural language preprocessing technology is used to extract the instructions and the user's emotional information in the obtained text data to obtain a text feature vector; the convolutional neural network technology is used to extract the user's facial features and / or lip shape change features in the user's image data in the user data to obtain an image feature vector; the neural network model technology is used to extract the features of the user's environment data in the user data to obtain an environment feature vector; the machine learning model technology is used to extract the user's behavior pattern features from the user's behavior data in the user data to obtain a behavior vector feature; the machine learning model technology is used to extract the user's device behavior features from the user's device information in the user data to obtain a device vector feature;

[0107] The text feature vector, image feature vector, environment feature vector, behavior vector feature and device vector feature are integrated to obtain a comprehensive feature vector;

[0108] A hybrid model is used to extract deep features from the comprehensive feature vector to obtain an intermediate comprehensive feature vector. The hybrid model is a model that combines a convolutional neural network and a long short-term memory network. A feature selection algorithm is used to remove redundant features in the intermediate comprehensive feature vector to obtain a final comprehensive feature vector. The feature selection algorithm includes a random forest algorithm and a decision tree.

[0109] The hybrid model is used to classify and identify the final comprehensive feature vector to obtain the user's specific intention; according to the user's specific intention, the final control strategy is generated;

[0110] The final control strategy is used to control the corresponding intelligent devices to perform corresponding actions.

[0111] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0112] The present application embodiment also provides a control device for an intelligent device. It should be noted that the control device for the intelligent device of the present application embodiment can be used to execute the control method for an intelligent device provided by the present application embodiment. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been described will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0113] The following is an introduction to the control device of the smart device provided in the embodiment of the present application.

[0114] Figure 3 is a structural block diagram of a control device for an intelligent device provided according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0115] The acquisition unit 31 is used to acquire user data, wherein the user data at least includes the user's voice data, the user's image data, and the user's environment data, wherein the user's environment data includes the temperature, humidity, and light intensity of the user's environment, and the user's image data includes the user's facial image and / or the user's lip shape change image;

[0116] A first processing unit 32 is used to extract features from each type of data in the above user data to obtain a comprehensive feature vector;

[0117] A second processing unit 33 is used to process the comprehensive feature vector using at least a convolutional neural network technology and a long short-term memory network technology to obtain a final control strategy;

[0118] The third processing unit 34 is used to control the corresponding intelligent device to perform corresponding actions by adopting the above-mentioned final control strategy.

[0119] In the above steps, by taking voice data, image data, and user environment data into consideration, the accuracy of understanding user intentions in the present application is improved compared to the existing solutions. Feature extraction is performed on each type of data in the above user data to obtain a comprehensive feature vector. The above comprehensive feature vector is processed by at least convolutional neural network technology and long short-term memory network technology to obtain the final control strategy. Compared with the existing solutions, the user's true intentions can be understood from multiple levels, thereby improving the accuracy of controlling smart devices, thereby solving the problem that the existing solutions control smart devices only according to the user's voice, resulting in the existing solutions The results of controlling smart devices cannot meet the real needs of users, which in turn leads to the low accuracy of the existing solutions in controlling smart devices.

[0120] In one embodiment of the present application, the first processing unit includes a first processing module, a second processing module, a third processing module and a fourth processing module.

[0121] The first processing module is used to convert the voice data of the user in the user data into text, and then extract the instructions and the emotion information of the user in the obtained text data by using natural language preprocessing technology to obtain a text feature vector;

[0122] The second processing module is used for extracting the facial features and / or lip shape change features of the user in the image data of the user in the user data by using convolutional neural network technology to obtain an image feature vector;

[0123] The third processing module is used to extract the characteristics of the user's environment data from the user data by using a neural network model technology to obtain an environment feature vector;

[0124] The fourth processing module is used to fuse the text feature vector, the image feature vector and the environment feature vector to obtain the comprehensive feature vector.

[0125] In one embodiment of the present application, the second processing unit includes a fifth processing module and a first generating module;

[0126] The fifth processing module is used to extract deep-level features from the comprehensive feature vector using a hybrid model to obtain a final comprehensive feature vector, wherein the hybrid model is a model combining a convolutional neural network and a long short-term memory network, and the deep-level features are features of a preset level;

[0127] The first generating module is used to generate the above-mentioned final control strategy according to the final comprehensive feature vector.

[0128] In one embodiment of the present application, the second processing unit includes a sixth processing module, a seventh processing module, and a second generating module;

[0129] The sixth processing module is used to extract deep features from the comprehensive feature vector using a hybrid model to obtain an intermediate comprehensive feature vector, wherein the hybrid model is a model combining a convolutional neural network and a long short-term memory network;

[0130] The seventh processing module is used to remove redundant features in the intermediate comprehensive feature vector by using a feature selection algorithm to obtain a final comprehensive feature vector, wherein the feature selection algorithm includes a random forest algorithm and a decision tree;

[0131] The second generating module is used to generate the above-mentioned final control strategy according to the above-mentioned final comprehensive feature vector.

[0132] In one embodiment of the present application, the first generation module or the second generation module includes a processing submodule and a generation submodule;

[0133] The processing submodule is used to classify and identify the final comprehensive feature vector using the hybrid model to obtain the specific intention of the user;

[0134] The generation submodule is used to generate the above-mentioned final control strategy according to the above-mentioned specific intention of the above-mentioned user.

[0135] In one embodiment of the present application, the above-mentioned user data also includes the behavior data of the above-mentioned user, and the above-mentioned behavior data of the above-mentioned user includes the number of steps, heart rate and activity time of the above-mentioned user. The above-mentioned device also includes a fourth processing unit, and the fourth processing unit is used to extract the user's behavior pattern characteristics from the above-mentioned behavior data of the above-mentioned user in the above-mentioned user data using machine learning model technology in the process of performing feature extraction on each type of data in the above-mentioned user data to obtain a comprehensive feature vector, so as to obtain a behavior vector feature.

[0136] In one embodiment of the present application, the above-mentioned user data also includes the device information of the above-mentioned user, and the above-mentioned device information of the above-mentioned user includes the chat records and calendar events of the above-mentioned user. The above-mentioned device also includes a fifth processing unit, and the fifth processing unit is used to extract the user's device behavior characteristics from the above-mentioned device information of the above-mentioned user in the above-mentioned user data using machine learning model technology in the process of performing feature extraction on each type of data in the above-mentioned user data to obtain a comprehensive feature vector, so as to obtain a device vector feature.

[0137] The control device of the above-mentioned intelligent device includes a processor and a memory, and the above-mentioned acquisition unit, the first processing unit, the second processing unit and the third processing unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0138] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the existing solution controls the smart device only according to the user's voice, resulting in the existing solution's control of the smart device being unable to meet the user's real needs, which in turn leads to the low accuracy of the existing solution in controlling the smart device.

[0139] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0140] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the control method of the smart device.

[0141] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the control method of the smart device is executed when the program is run.

[0142] The embodiment of the present invention provides a device, which includes a processor, a memory, and a program stored in the memory and can be run on the processor. When the processor executes the program, it implements at least the following steps: obtaining user data, the user data at least includes the user's voice data, the user's image data, and the user's environment data, the user's environment data includes the temperature, humidity and light intensity of the user's environment, and the user's image data includes the user's facial image and / or the user's lip shape change image; extracting features from each type of data in the user data to obtain a comprehensive feature vector; processing the comprehensive feature vector using at least convolutional neural network technology and long short-term memory network technology to obtain a final control strategy; using the final control strategy to control the corresponding intelligent device to perform corresponding actions. The device in this article can be a server, PC, PAD, mobile phone, etc.

[0143] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with at least the following method steps: obtaining user data, the user data at least including the user's voice data, the user's image data, and the user's environment data, the user's environment data including the temperature, humidity and light intensity of the user's environment, and the user's image data including the user's facial image and / or the user's lip shape change image; performing feature extraction on each type of data in the user data to obtain a comprehensive feature vector; processing the comprehensive feature vector using at least convolutional neural network technology and long short-term memory network technology to obtain a final control strategy; and using the final control strategy to control the corresponding intelligent device to perform corresponding actions.

[0144] The present application also provides a control system for an intelligent device, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the above methods. By taking voice data, image data, and user environment data into consideration, the present application improves the accuracy of understanding the user's intention compared to the existing solutions, extracts features of each type of data in the above user data, obtains a comprehensive feature vector, and processes the above comprehensive feature vector using at least convolutional neural network technology and long short-term memory network technology to obtain a final control strategy. Compared with the existing solutions, the real intention of the user can be understood from multiple levels, thereby improving the accuracy of controlling the intelligent device, thereby solving the problem that the existing solutions only control the intelligent device according to the user's voice, resulting in the inability of the existing solutions to control the intelligent device to meet the real needs of the user, thereby resulting in the low accuracy of the existing solutions to control the intelligent device.

[0145] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0146] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0152] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0154] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0155] 1) The control method of the smart device of the present application takes voice data, image data, and user environment data into consideration, thereby improving the accuracy of understanding the user's intention compared to the existing solutions. Feature extraction is performed on each type of data in the above user data to obtain a comprehensive feature vector. The above comprehensive feature vector is processed by at least convolutional neural network technology and long short-term memory network technology to obtain a final control strategy. Compared with the existing solutions, the true intention of the user can be understood from multiple levels, thereby improving the accuracy of controlling the smart device, thereby solving the problem that the existing solutions control the smart device only according to the user's voice, resulting in the inability of the existing solutions to control the smart device to meet the real needs of the user, thereby resulting in the low accuracy of the existing solutions to control the smart device.

[0156] 2) The control device of the smart device of the present application takes voice data, image data, and user environment data into consideration, thereby improving the accuracy of understanding the user's intention compared to the existing solutions. Feature extraction is performed on each type of data in the above user data to obtain a comprehensive feature vector. The above comprehensive feature vector is processed by at least convolutional neural network technology and long short-term memory network technology to obtain a final control strategy. Compared with the existing solutions, the real intention of the user can be understood from multiple levels, thereby improving the accuracy of controlling the smart device, thereby solving the problem that the existing solutions control the smart device only according to the user's voice, resulting in the inability of the existing solutions to control the smart device to meet the real needs of the user, thereby resulting in the low accuracy of the existing solutions to control the smart device.

[0157] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A control method for an intelligent device, characterized in that: include: Acquire user data, the user data at least including user voice data, user image data, and user environment data, the user environment data including temperature, humidity, and light intensity of the user environment, and the user image data including the user's facial image and / or the user's lip shape change image; Extracting features of various types of data in the user data to obtain a comprehensive feature vector; At least using convolutional neural network technology and long short-term memory network technology to process the comprehensive feature vector to obtain a final control strategy; The final control strategy is used to control the corresponding intelligent device to perform corresponding actions.

2. The method according to claim 1, characterized in that: Perform feature extraction on each type of data in the user data to obtain a comprehensive feature vector, including: After the voice data of the user in the user data is converted into text, the instructions and the emotion information of the user in the obtained text data are extracted using natural language preprocessing technology to obtain a text feature vector; Using convolutional neural network technology to extract facial features and / or lip shape change features of the user in the image data of the user in the user data to obtain an image feature vector; Using a neural network model technology to extract features of the user's environment data from the user data to obtain an environment feature vector; The text feature vector, the image feature vector and the environment feature vector are fused to obtain the comprehensive feature vector.

3. The method according to claim 1, characterized in that At least convolutional neural network technology and long short-term memory network technology are used to process the comprehensive feature vector to obtain a final control strategy, including: A hybrid model is used to extract deep features from the comprehensive feature vector to obtain a final comprehensive feature vector, wherein the hybrid model is a model combining a convolutional neural network and a long short-term memory network, and the deep features are features of a preset level; The final control strategy is generated according to the final comprehensive feature vector.

4. The method according to claim 1, characterized in that: At least convolutional neural network technology and long short-term memory network technology are used to process the comprehensive feature vector to obtain a final control strategy, including: A hybrid model is used to extract deep features from the comprehensive feature vector to obtain an intermediate comprehensive feature vector, wherein the hybrid model is a model combining a convolutional neural network and a long short-term memory network; Using a feature selection algorithm to remove redundant features in the intermediate comprehensive feature vector to obtain a final comprehensive feature vector, the feature selection algorithm including a random forest algorithm and a decision tree; The final control strategy is generated according to the final comprehensive feature vector.

5. The method according to claim 3 or 4, characterized in that: Generating the final control strategy according to the final comprehensive feature vector includes: Using the hybrid model to classify and identify the final comprehensive feature vector to obtain the specific intention of the user; The final control strategy is generated according to the specific intention of the user.

6. The method according to claim 1, characterized in that The user data also includes the user's behavior data, and the user's behavior data includes the number of steps, heart rate, and activity time of the user. In the process of extracting features from each type of data in the user data to obtain a comprehensive feature vector, the method further includes: The machine learning model technology is used to extract the user's behavior pattern characteristics from the behavior data of the user in the user data to obtain the behavior vector characteristics.

7. The method according to claim 1, characterized in that The user data also includes the device information of the user, and the device information of the user includes the chat records and calendar events of the user. In the process of extracting features of various types of data in the user data to obtain a comprehensive feature vector, the method further includes: The machine learning model technology is used to extract the user's device behavior characteristics from the user's device information in the user data to obtain device vector characteristics.

8. A control device for an intelligent device, characterized in that: include: an acquisition unit, configured to acquire user data, wherein the user data at least includes user voice data, user image data, and user environment data, wherein the user environment data includes temperature, humidity, and light intensity of the user environment, and the user image data includes a facial image of the user and / or an image of the user's lip shape changes; A first processing unit, configured to extract features from each type of data in the user data to obtain a comprehensive feature vector; A second processing unit is used to process the comprehensive feature vector using at least a convolutional neural network technology and a long short-term memory network technology to obtain a final control strategy; The third processing unit is used to control the corresponding intelligent device to perform corresponding actions by adopting the final control strategy.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A control system for an intelligent device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images