Equipment control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a device control method and device, an electronic device and a storage medium. The method comprises the steps that emotion data collected by a first terminal device for a target user is acquired; inputting the emotion data into a target neural network model, and outputting emotional state information of the target user, the target neural network model being a neural network model for performing emotional state recognition in combination with a neural network algorithm and an attention mechanism; and controlling one or more second terminal devices based on the emotional state information. According to the embodiment of the invention, the terminal equipment can be controlled according to the identified emotional state of the user, a personalized server is provided for the user, and the use experience of the user equipment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of device control, and in particular to a device control method and apparatus, electronic equipment, and storage medium. Background Art

[0002] With the rapid development of the Internet of Things and artificial intelligence technologies, smart home systems have become an important part of modern families. However, most existing smart home systems are based on fixed preset rules or manual user control to achieve device linkage. This makes it difficult for the system to provide personalized services based on the actual needs of users, reducing user experience. Summary of the invention

[0003] In view of the above problems, a device control method and apparatus, electronic device, and storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0004] A device control method, the method comprising:

[0005] Acquiring emotion data collected by the first terminal device for the target user;

[0006] The emotional data is input into a target neural network model, and the emotional state information of the target user is output, wherein the target neural network model is a neural network model that combines a neural network algorithm and an attention mechanism to perform emotional state recognition;

[0007] One or more second terminal devices are controlled based on the emotional state information.

[0008] Optionally, controlling one or more second terminal devices based on the emotional state information includes:

[0009] Obtaining device status information of one or more second terminal devices;

[0010] generating a device control strategy for the one or more second terminal devices according to the device state information and the emotional state information;

[0011] The one or more second terminal devices are controlled according to the device control strategy.

[0012] Optionally, generating a device control strategy for the one or more second terminal devices according to the device state information and the emotional state information includes:

[0013] Obtain a reinforcement learning algorithm that optimizes and generates device control strategies based on historical device states and historical emotional states;

[0014] The device state information and the emotional state information are input into the reinforcement learning algorithm to generate an optimized device control strategy.

[0015] Optionally, it also includes:

[0016] Obtaining feedback information from the target user regarding the device control strategy;

[0017] The target neural network model and / or the reinforcement learning algorithm are adjusted based on the feedback information.

[0018] Optionally, it also includes:

[0019] When the environmental state of the second terminal device is updated, the target neural network model and / or the reinforcement learning algorithm are adjusted based on the updated environmental state.

[0020] Optionally, it also includes:

[0021] Obtaining feedback information from the target user regarding the device control strategy;

[0022] Acquiring device usage data of the target user for the one or more second terminal devices;

[0023] The device linkage strategy is updated based on the feedback information and the device usage data.

[0024] Optionally, before inputting the emotion data into the target neural network model, the method further includes:

[0025] The emotional data is preprocessed, wherein the preprocessing includes any one or more of pre-emphasis processing, normalization processing, and endpoint detection.

[0026] Optionally, the target neural network model is a convolutional neural network model, and the fully connected layer in the convolutional neural network model is connected to an attention layer for executing an attention mechanism, and the attention layer is used to receive feature data output by the fully connected layer, and use the attention mechanism to extract target emotional features for representing the emotional state of the target user from the model data.

[0027] Optionally, the training process of the target neural network model includes the following steps:

[0028] determining an initial neural network model from a plurality of pre-trained neural network models;

[0029] Transfer learning is performed based on the initial neural network model to obtain a target neural network model.

[0030] A device control apparatus, comprising:

[0031] An emotion data acquisition module, used to acquire emotion data collected by the first terminal device for a target user;

[0032] An emotional state determination module, used to input the emotional data into a target neural network model and output emotional state information of the target user, wherein the target neural network model is a neural network model that combines a neural network algorithm and an attention mechanism to perform emotional state recognition;

[0033] A device control module is used to control one or more second terminal devices based on the emotional state information.

[0034] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the device control method as described above when executed by the processor.

[0035] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the device control method described above is implemented.

[0036] The embodiments of the present invention have the following advantages:

[0037] In an embodiment of the present invention, emotional data collected by a first terminal device for a target user can be obtained; the emotional data is input into a target neural network model, and then the emotional state information of the target user is output, and then one or more second terminal devices can be controlled based on the emotional state information, and then the terminal device can be controlled according to the identified emotional state of the user, providing the user with a personalized server and improving the user's device usage experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0039] Figure 1 is a flowchart of a device control method provided by an embodiment of the present invention;

[0040] Figure 2 is a flowchart of another device control method provided by an embodiment of the present invention;

[0041] Figure 3 is a flowchart of another device control method provided by an embodiment of the present invention;

[0042] Figure 4is a flowchart of another device control method provided by an embodiment of the present invention;

[0043] Figure 5 It is a structural schematic diagram of a device control system provided by an embodiment of the present invention;

[0044] Figure 6 It is a structural schematic diagram of a device control apparatus provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Reference Figure 1 , shows a flow chart of a device control method provided by an embodiment of the present invention, which may specifically include the following steps:

[0047] Step S101, acquiring emotion data collected by a first terminal device for a target user;

[0048] The first terminal device may be provided with a device for collecting emotion data sensors. In the embodiment of the present invention, the emotion data may include but is not limited to the user's voice data, facial data, body movement data, and other data that can be used to characterize the user's emotions. For example, the sound produced when a user cries or laughs is different, so the user's current emotional state can be reversely deduced by obtaining the user's voice data.

[0049] In one example, the facial data and body movement data of the target user may be determined using images or videos recorded by a camera, that is, the first terminal device may be a device carrying a camera.

[0050] In one example, a microphone may be used to collect sounds in the environment, that is, the first terminal device may be a device carrying a microphone.

[0051] In one embodiment of the present invention, after collecting emotional data, since the emotional data collected in the environment contains a lot of environmental interference, in order to more accurately identify the emotional state corresponding to the emotional data, the emotional data can be preprocessed before inputting the emotional data into the target neural network model. By preprocessing the emotional data, the environmental interference in the emotional data can be filtered out.

[0052] In an example, the preprocessing may include any one or more of pre-emphasis processing, normalization processing, and endpoint detection.

[0053] Taking the emotional data as speech data as an example, the above preprocessing process is explained:

[0054] Pre-emphasis processing is to enhance the high-frequency signal and compensate for the high-frequency attenuation of the voice signal during transmission. In practical applications, a high-pass filter can be used to implement pre-emphasis processing, and the filter can be expressed as a first-order difference equation; normalization processing can eliminate the influence of volume differences between different speakers, so that the energy of all voice signals is within a similar range. The range can be set according to the actual scenario, and there is no restriction on this in the embodiment of the present invention. In practical applications, the normalization method may include amplitude normalization and energy normalization. Amplitude normalization is to scale the amplitude of the signal to a fixed range, such as [-1,1]; energy normalization is to adjust the energy of the signal to a fixed level. Endpoint detection is to identify the starting point and end point of the voice signal, remove useless silent segments, and reduce the amount of calculation. In practical applications, endpoint detection methods may include methods based on energy thresholds, methods based on zero-crossing rates, and composite methods combining energy and zero-crossing rates.

[0055] By preprocessing the sentiment data using methods such as pre-emphasis, normalization, and endpoint detection, the input quality of the target neural network model can be improved.

[0056] Step S102, inputting the emotion data into a target neural network model, and outputting the emotional state information of the target user, wherein the target neural network model is a neural network model that combines a neural network algorithm and an attention mechanism to perform emotional state recognition;

[0057] The emotional state information may include but is not limited to the target user's emotions such as happiness, sadness, anger, etc.

[0058] In an embodiment of the present invention, a target neural network model may be pre-trained, the input data of the neural network model being the user's emotion data, and the output data being the user's emotional state information.

[0059] In practical applications, after acquiring the emotion data, the emotion data can be output to the target neural network model for emotion recognition, and the user's current emotional state information analyzed by the target neural network model can be input to realize terminal device control based on the emotional state data.

[0060] In an embodiment of the present invention, a neural network algorithm can be set in the target neural network model, and the neural network algorithm can be combined with an attention mechanism. Thus, by introducing the attention mechanism in the target neural network model, the emotional feature extraction capability of the neural network model can be improved, thereby making the final identified emotional state more accurate.

[0061] In one embodiment of the present invention, the type of the target neural network model depends on the specific application scenario and task requirements. There are many types of neural network models, for example, feedforward neural networks, convolutional neural networks, recurrent neural networks, long short-term memory networks, gated recurrent units, autoencoders, variational autoencoders, generative adversarial networks, graph neural networks, reinforcement learning networks, etc.

[0062] In one embodiment of the present invention, the target neural network model may be a convolutional neural network model, and the neural network algorithm may be a CNN (Convolutional Neural Network) algorithm, wherein the neural network model may include but is not limited to the following parts:

[0063] Input layer: receives raw data, such as image data or speech data.

[0064] Convolutional layer: Multiple convolutional layers, each containing multiple convolutional kernels. The convolutional layer is the core part of CNN and is used to extract local features of input data. Each convolutional layer contains multiple convolutional kernels (or filters), which slide over the input data, calculate the dot product of the local area, and generate a feature map. The convolution operation can capture the spatial hierarchical structure of the input data, such as edges, textures, etc.

[0065] Activation function: After the convolution layer, a nonlinear activation function such as ReLU (Rectified Linear Unit) can be applied to introduce nonlinear characteristics and enhance the expressiveness of the model.

[0066] Pooling layer: Pooling operation is applied after each convolutional layer. Pooling layer is used to reduce the spatial dimension (height and width) of feature map, reduce the amount of computation, and enhance the translation invariance of features. Pooling operation can include maximum pooling and average pooling.

[0067] Fully connected layer: One or more fully connected layers are used to integrate features and classify. After passing through multiple convolutional layers and pooling layers, the feature map is flattened into a one-dimensional vector and then sent to the fully connected layer. The fully connected layer is used to integrate and classify these features. It can contain multiple neurons, each of which is connected to all neurons in the previous layer.

[0068] Output layer: softmax layer, used to output classification results. The output layer can be a fully connected layer, and different activation functions can be used depending on the task. For example, for classification tasks, the output layer can use the softmax activation function to convert the output into a probability distribution.

[0069] In an embodiment of the present invention, the fully connected layer in the convolutional neural network model is connected to the attention layer for executing the attention mechanism. The attention layer is used to receive the feature data output by the fully connected layer, and use the attention mechanism to extract the target emotional features for representing the emotional state of the target user from the model data.

[0070] By setting an attention layer after the fully connected layer, the attention layer can apply the attention mechanism. The attention mechanism can extract the parts of the features that are most relevant to emotional expression (such as certain syllables or words in voice data, specific areas of the face in image data, eyes, and mouth), so that the target neural network model can accurately identify the user's emotional state when processing real-time input voice and image data.

[0071] In the embodiments of the present invention, based on the characteristics of emotion recognition, basic neural network models such as CNN and RNN (Recurrent Neural Network) can be improved, such as introducing an attention mechanism to improve the target neural network model's ability to recognize key features.

[0072] In one embodiment of the present invention, the training process of the target neural network model includes the following steps: determining an initial neural network model from a plurality of pre-trained neural network models; performing transfer learning based on the initial neural network model to obtain a target neural network model.

[0073] In practical applications, transfer learning is a machine learning method that allows the knowledge learned on one task (such as a CNN model pre-trained on a large-scale dataset) to be used to help solve another related task (such as a specific emotion recognition task). This method is particularly suitable for situations where the target task has a small amount of data, because it can reduce the need for a large amount of labeled data.

[0074] In an embodiment of the present invention, the target neural network model can be obtained by applying transfer learning training, and its training process can be divided into: a pre-training model selection process and a pre-training model fine-tuning process using transfer learning.

[0075] First, select a CNN model (initial neural network model) pre-trained on a large-scale dataset (such as ImageNet). These models are models that have learned how to extract useful image features.

[0076] In the process of fine-tuning the initial neural network model using transfer learning, the convolutional layers of the initial neural network model can be frozen because these layers have learned general image feature extraction capabilities. Freezing these layers means that their weights will not be updated in subsequent training.

[0077] The fully connected layers of the initial neural network model are designed for the original task (such as image classification). In the emotion recognition task, these layers may no longer be applicable, so they need to be replaced with new fully connected layers to make them more suitable for the target task (i.e. emotion recognition).

[0078] In the embodiment of the present invention, only the newly replaced fully connected layer can be trained, while the weight of the convolutional layer remains unchanged, so that the feature extraction capability of the pre-trained model can be utilized while adapting the model to new tasks.

[0079] In one embodiment of the present invention, some convolutional layers may be gradually unfrozen and fine-tuned so that the model can better adapt to a specific task. Specifically, some convolutional layers may be gradually unfrozen and fine-tuned when the target neural network model generated based on the training of the initial neural network model performs poorly on a new task.

[0080] In the embodiment of the present invention, the use of transfer learning to train the target neural network model has the following advantages:

[0081] (1) Reduce data dependence: By leveraging the feature extraction capabilities of pre-trained models, the need for large amounts of labeled data can be reduced.

[0082] (2) Accelerated training: Since most weights have been pre-trained, the training speed can be accelerated.

[0083] (3) Improved performance: Pre-trained models usually have strong feature extraction capabilities, which helps improve the performance of new tasks.

[0084] In the embodiment of the present invention, the effect of transfer learning depends largely on the correlation between the source task and the target task. If the task difference is large, the effect of transfer learning may not be good. And when the amount of data is small, excessive fine-tuning may lead to overfitting of the model.

[0085] In one embodiment of the present invention, for the multi-label problem of emotion recognition, multi-label classification loss functions such as Hinge loss and Logistic loss can be used to optimize the objective function of the target neural network model.

[0086] In the embodiment of the present invention, the learning rate can be dynamically adjusted according to the training progress and performance of the target neural network model to improve the convergence speed and stability of the model. Then, the prediction results of multiple models are integrated to improve the accuracy of the overall prediction through voting or weighted voting.

[0087] Step S103: controlling one or more second terminal devices based on the emotional state information.

[0088] The second terminal device may include the first terminal device, or may be another terminal device other than the first terminal device.

[0089] After determining the emotional state information of the target user, one or more second terminal devices can be controlled according to the emotional state information, so that the one or more second terminal devices can perform corresponding actions according to the emotional state information, such as adjusting the light color, playing music, adjusting the indoor temperature, etc.

[0090] By performing corresponding actions through one or more second terminal devices, the current scene can be adapted to the target user. For example, when it is recognized that the user's current emotion is sad, soothing music can be played and the lights can be adjusted to soft warm colors.

[0091] In one example, the second terminal device is a device associated with the current location of the target user. Step S103 may specifically be: determining the device type based on the emotional state, determining the target second terminal device according to the device type and the location of the target user, and then generating a control instruction for the target second terminal device based on the emotional state information, and sending the control instruction to the target second terminal device.

[0092] For example, when it is determined that the lights need to be turned on and music needs to be played based on the target user's current mood, the lights and music playback devices at which specific locations should be turned on can be determined based on the target user's location. In actual applications, the lamps in the user's space can be turned on, that is, when the user is currently in the bedroom, the bedroom lamps can be turned on; music playback devices are less distributed than lamps, so the music playback device closest to the target user can be started according to the distance between the music playback device and the user, or a startable range can be set for the music playback device, and the music playback devices within the start-up range can be turned on; when there are multiple music playback devices within the start-up range, the nearest music playback device can be selected according to the distance, or a device can be selected based on the characteristics of each music playback device.

[0093] In one embodiment of the present invention, the first terminal device and the second terminal device can be connected to the central controller, and the central controller can control each terminal device through a network with a preset protocol. In the central controller, control instructions for one or more terminals can be generated in combination with the emotional state information output by the target neural network model, and then the control instructions can be sent to the second terminal device using the preset protocol to control the second terminal device to execute the control instructions.

[0094] Among them, the preset protocol can be the SigMesh protocol, which is a mesh network protocol based on Bluetooth low energy technology. It has the characteristics of low power consumption, scalability, security and reliability, and is suitable for various application scenarios such as smart home, building automation, and industrial automation. Through multi-hop communication and multi-layer security mechanisms, the SigMesh protocol can provide an efficient and stable wireless communication solution.

[0095] In an embodiment of the present invention, emotional data collected by a first terminal device for a target user can be obtained; the emotional data is input into a target neural network model to output emotional state information of the target user; one or more second terminal devices are controlled based on the emotional state information, and then the terminal devices can be controlled according to the identified emotional state of the user, providing the user with a personalized server and improving the user's device usage experience.

[0096] Reference Figure 2 , shows a flowchart of another device control method provided by an embodiment of the present invention, which may specifically include the following steps:

[0097] Step S201, obtaining emotion data collected by the first terminal device for the target user;

[0098] Step S202, inputting the emotion data into a target neural network model, and outputting the emotional state information of the target user, wherein the target neural network model is a neural network model that combines a neural network algorithm and an attention mechanism to perform emotional state recognition;

[0099] Step S203, obtaining device status information of one or more second terminal devices;

[0100] Among them, the equipment status information is multifaceted, including but not limited to basic information, operating status, performance parameters, faults and alarms, location and environment, network and communication, users and permissions, and historical data, etc. This information can be obtained through various sensors, controllers and communication interfaces, and the operation of the equipment can be monitored and optimized through data processing and analysis.

[0101] The basic information may include: device ID (unique identifier of the device), device name (name or label of the device), device type, firmware version, hardware version, etc.

[0102] The operating status may include: switch status, operating mode (such as automatic, manual, energy saving, etc.), working status (running, standby, fault, etc.), task status (if the device performs a specific task, the progress and status of the task), etc.

[0103] Performance parameters may include: temperature, humidity, voltage, current, power, energy consumption, etc.

[0104] Faults and alarms may include: fault codes, alarm information (relevant information when the equipment alarms, such as alarm type, alarm level, etc.), historical fault records, etc.

[0105] The location and environment may include: location information (the geographical location of the device, which may include latitude, longitude, altitude, etc.), environmental parameters (environmental parameters around the device, such as light intensity, air quality, etc.), etc.

[0106] Network and communication may include: network connection status, signal strength, communication delay, etc.

[0107] Users and permissions may include user information, users and permissions, etc.

[0108] Historical data may include: historical records, statistical data, etc.

[0109] Step S204, generating a device control strategy for one or more second terminal devices according to the device state information and the emotional state information;

[0110] After determining the setting state information and the emotional state information, a device control strategy matching the emotional state information may be generated, and the device control strategy may include execution actions of each second terminal device. The device control strategy may be used for linkage control of multiple devices.

[0111] Specifically, the process of generating the device control strategy is as follows: obtaining a reinforcement learning algorithm for optimizing the device control strategy based on historical device status and historical emotional status; inputting device status information and emotional status information into the reinforcement learning algorithm to generate an optimized device control strategy.

[0112] In practical applications, reinforcement learning (RL) algorithms can be used to learn how to adjust the control strategies of smart home devices according to the current emotional state and device status to optimize the user experience.

[0113] Among them, the reinforcement learning algorithm can be generated based on the historical device state and historical emotional state of the second terminal device. The reinforcement learning algorithm can obtain the optimal device control strategy for any setting state information and emotional state information.

[0114] In one example, reinforcement learning is used in adaptive learning as follows:

[0115] State space: Define a state space, including the user's emotional state, device state, historical usage data, etc.

[0116] Action space: Define an action space, including various control actions that can be performed on smart home devices.

[0117] Reward function: Design a reward function to evaluate the impact of each action on the user’s emotional state and comfort.

[0118] Learning process: The reinforcement learning agent learns to take the best action (i.e. the optimal device control strategy) in a given state to maximize the cumulative reward through interaction with the environment.

[0119] Step S205: Control one or more second terminal devices according to the device control strategy.

[0120] In actual applications, the central controller can generate a control instruction including a device control strategy, and send the control instruction to one or more second terminal devices. Then the second terminal device parses the device control strategy from the control instruction and executes the device control strategy.

[0121] In one embodiment of the present invention, the device linkage strategy can change according to changes in user needs. Specifically, feedback information of the target user on the device control strategy can be obtained; device usage data of the target user for one or more second terminal devices can be obtained; and the device linkage strategy can be updated based on the feedback information and device usage data.

[0122] In actual applications, user feedback information on the currently executed device control strategy and actual device usage data of the second terminal device can be collected to update the formulated device linkage strategy.

[0123] In the embodiment of the present invention, the user preference can also be determined based on the device usage data of the target user on the second terminal device, and then a device linkage strategy that meets the user preference can be formulated. In the subsequent use process, the device linkage strategy can be continuously updated in combination with user feedback information and changes in user preferences, thereby ensuring that the device linkage strategy meets user needs, realizing personalized device control, and improving the user's device usage experience.

[0124] In the embodiment of the present invention, a device linkage strategy is generated according to the recognized emotional state of the user, and then multiple second terminal devices can be controlled according to the device linkage strategy, providing a personalized server for the user and improving the user's device usage experience.

[0125] Reference Figure 3 , shows a flowchart of another device control method provided by an embodiment of the present invention, which may specifically include the following steps:

[0126] Step S301, acquiring emotion data collected by the first terminal device for the target user;

[0127] Step S302, inputting the emotion data into a target neural network model, and outputting the emotional state information of the target user, wherein the target neural network model is a neural network model that combines a neural network algorithm and an attention mechanism to perform emotional state recognition;

[0128] Step S303, obtaining device status information of one or more second terminal devices;

[0129] Step S304, obtaining a reinforcement learning algorithm for optimizing the device control strategy based on the historical device state and the historical emotional state;

[0130] Step S305: input the device state information and the emotional state information into a reinforcement learning algorithm to generate an optimized device control strategy.

[0131] Step S306: Control one or more second terminal devices according to the device control policy.

[0132] Step S307, obtaining feedback information of the target user on the device control strategy;

[0133] In actual applications, questionnaires can be sent to users regularly to collect their feedback on system performance, interface friendliness, functional satisfaction, etc., and the customer support team can collect problems, suggestions and complaints encountered by users during use.

[0134] Step S308: adjusting the target neural network model and / or reinforcement learning algorithm based on the feedback information.

[0135] After obtaining user feedback information, the target neural network model and the reinforcement learning algorithm can be adjusted and optimized simultaneously according to the feedback information, or the target neural network model can be adjusted alone according to the feedback information, or the reinforcement learning algorithm can be adjusted according to the feedback information.

[0136] In an embodiment of the present invention, the target neural network model and / or reinforcement learning algorithm can be adjusted based on the user's feedback information on the device control strategy to avoid the target neural network model and / or reinforcement learning algorithm from becoming outdated, and personalized servers can be continuously provided to users to improve the user's device usage experience.

[0137] Reference Figure 4 , shows a flowchart of another device control method provided by an embodiment of the present invention, which may specifically include the following steps:

[0138] Step S401, obtaining emotion data collected by the first terminal device for the target user;

[0139] Step S402, inputting the emotion data into a target neural network model, and outputting the emotional state information of the target user, wherein the target neural network model is a neural network model that combines a neural network algorithm and an attention mechanism to perform emotional state recognition;

[0140] Step S403, obtaining device status information of one or more second terminal devices;

[0141] Step S404, obtaining a reinforcement learning algorithm for optimizing the device control strategy based on the historical device state and the historical emotional state;

[0142] Step S405: input the device state information and the emotional state information into a reinforcement learning algorithm to generate an optimized device control strategy.

[0143] Step S406: Control one or more second terminal devices according to the device control policy.

[0144] Step S407: When the environmental state of the second terminal device is updated, the target neural network model and / or reinforcement learning algorithm is adjusted based on the updated environmental state.

[0145] In actual applications, there will be certain differences when different terminal devices are used in different environments. When the environmental state of the terminal device is updated, the previous target neural network model and / or reinforcement learning algorithm may not be incompatible with the current environmental state of the second terminal device. Then, the target neural network model and / or reinforcement learning algorithm can be adjusted according to the updated environmental state to make the target neural network model and / or reinforcement learning algorithm adapt to the second terminal device.

[0146] In an embodiment of the present invention, the target neural network model and / or reinforcement learning algorithm can be adjusted according to the environmental status update of the second terminal device to avoid the target neural network model and / or reinforcement learning algorithm from becoming outdated, and personalized servers can be continuously provided to users to improve the user device usage experience.

[0147] Reference Figure 5, is a device control system in an embodiment of the present invention. The device control system may include an adaptive module, a central controller, an emotion recognition module and a smart home device. The adaptive module and the central controller communicate via the SigMesh protocol, and the emotion recognition module and the central controller communicate via the SigMesh protocol. The SigMesh network is the communication basis of the entire system. Smart home devices communicate with each other via the SigMesh network to share data and control instructions. Smart home devices communicate with the central controller via the SigMesh protocol, and smart home devices may include multiple terminal devices. Figure 5 The architecture in can realize the linkage control of multiple smart home devices based on the recognized emotions.

[0148] Specifically, the functions of each part are as follows:

[0149] The emotion recognition module captures the user's voice, expression, behavior and other emotional data in real time through sensors (such as cameras, microphones, etc.) built into smart home devices, and identifies the user's emotional state through algorithms such as deep learning. Among them, a convolutional neural network algorithm model can be specifically set in the emotion recognition module, and the convolutional neural network algorithm (CNN) model can be optimized by introducing an attention mechanism. In the emotion recognition module, an attention layer can be added after the fully connected layer of CNN to extract the most relevant parts of the features for emotional expression (such as certain syllables or words in voice data, specific areas of the face in image data, eyes, and mouth), so that the model can accurately identify the user's emotional state when processing real-time input voice and image data, and provide timely input for the adaptive learning module.

[0150] In the emotion recognition module, the speech data can be pre-processed by using methods such as pre-emphasis, normalization, and endpoint detection to improve the input quality of the convolutional neural network algorithm model. Then, according to the characteristics of emotion recognition, the attention mechanism is introduced into the convolutional neural network algorithm model to improve the recognition ability of the convolutional neural network algorithm model for key features. In addition, during the training process of the convolutional neural network algorithm model, the multi-label classification loss functions such as Hinge loss and Logistic loss can be used to optimize the objective function of the model for the multi-label problem of emotion recognition.

[0151] For the optimization of adaptive learning algorithms and models, the learning rate can be adjusted dynamically according to the training progress and performance of the model to improve the convergence speed and stability of the model. Then, the prediction results of multiple models are integrated to improve the accuracy of the overall prediction through voting or weighted voting.

[0152] The adaptive learning module can receive the output results of the emotion recognition module (i.e., emotional state information) and the device state information of smart home devices, and use machine learning algorithms (such as reinforcement learning algorithms) to process and analyze these data to automatically adjust the device linkage strategy of home devices. In this way, the system can provide a more intelligent and personalized home experience based on the user's emotional state and needs.

[0153] The central controller is responsible for receiving the device linkage strategy generated by the adaptive learning module and sending control instructions to each smart home device through the SigMesh network. At the same time, the central controller can also monitor and manage the operating status of the entire system to ensure the stable operation of the system.

[0154] In order to maintain the healthy operation of the system in the embodiment of the present invention, the algorithms in the adaptive learning module and the emotion recognition module can be updated regularly. Specifically, on the one hand, questionnaires can be sent to users regularly to collect their feedback on system performance, interface friendliness, functional satisfaction, etc., and the customer support team can collect problems, suggestions and complaints encountered by users during use to determine user feedback information. On the other hand, data analysis tools (such as SPSS, R language, etc.) can be used to analyze user behavior data to discover user usage habits and preferences. Then, combined with user feedback information and user usage habits and preferences, the specific needs of users in different scenarios are identified. According to the identified needs, linkage strategies are formulated or updated to ensure that the system can provide the best user experience in different scenarios. For example, the recommended content or interaction method is adjusted according to the user's emotional changes, and the interface layout or function order is automatically adjusted according to the user's usage habits.

[0155] After optimizing the algorithm, the optimized algorithm, model and linkage strategy can be deployed to the system to ensure that users can immediately experience the improved functions.

[0156] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0157] Reference Figure 6 , shows a schematic diagram of the structure of a device control device provided by an embodiment of the present invention, which may specifically include the following modules:

[0158] The emotion data acquisition module 601 is used to acquire the emotion data collected by the first terminal device for the target user;

[0159] Emotional state determination module 602, used to input the emotional data into a target neural network model and output the emotional state information of the target user, wherein the target neural network model is a neural network model that combines a neural network algorithm and an attention mechanism to perform emotional state recognition;

[0160] The device control module 603 is used to control one or more second terminal devices based on the emotional state information.

[0161] In one embodiment of the present invention, the device control module 603 may include:

[0162] A device status information acquisition submodule, used to acquire device status information of one or more second terminal devices;

[0163] A device control strategy generating submodule, configured to generate a device control strategy for the one or more second terminal devices according to the device state information and the emotional state information;

[0164] The device control submodule is used to control the one or more second terminal devices according to the device control strategy.

[0165] In one embodiment of the present invention, the device control strategy generation submodule may include:

[0166] A reinforcement learning algorithm acquisition unit, used to acquire a reinforcement learning algorithm for optimizing a device control strategy based on historical device states and historical emotional states;

[0167] The device control strategy determination unit is used to input the device state information and the emotional state information into the reinforcement learning algorithm to generate an optimized device control strategy.

[0168] In one embodiment of the present invention, the device further comprises:

[0169] A first feedback information acquisition module, used to acquire feedback information of the target user regarding the device control strategy;

[0170] The first adjustment module is used to adjust the target neural network model and / or the reinforcement learning algorithm based on the feedback information.

[0171] In one embodiment of the present invention, the device further comprises:

[0172] The second adjustment module is used to adjust the target neural network model and / or the reinforcement learning algorithm based on the updated environmental state when the environmental state of the second terminal device is updated.

[0173] In one embodiment of the present invention, the device further comprises:

[0174] A second feedback information acquisition module, used to acquire feedback information of the target user regarding the device control strategy;

[0175] A device usage data acquisition module, used to acquire device usage data of the target user for the one or more second terminal devices;

[0176] A device linkage strategy updating module is used to update the device linkage strategy based on the feedback information and the device usage data.

[0177] In one embodiment of the present invention, the device further comprises:

[0178] A preprocessing module is used to preprocess the emotional data, wherein the preprocessing includes any one or more of pre-emphasis processing, normalization processing, and endpoint detection.

[0179] In one embodiment of the present invention, the target neural network model is a convolutional neural network model, and the fully connected layer in the convolutional neural network model is connected to the attention layer for executing the attention mechanism. The attention layer is used to receive the feature data output by the fully connected layer, and use the attention mechanism to extract the target emotional features for representing the emotional state of the target user from the model data.

[0180] In one embodiment of the present invention, the device further comprises:

[0181] An initial neural network model determination module, used to determine an initial neural network model from a plurality of pre-trained neural network models;

[0182] The transfer learning module is used to perform transfer learning based on the initial neural network model to obtain a target neural network model.

[0183] In an embodiment of the present invention, emotional data collected by a first terminal device for a target user can be obtained; the emotional data is input into a target neural network model, and then the emotional state information of the target user is output, and then one or more second terminal devices can be controlled based on the emotional state information, and then the terminal device can be controlled according to the identified emotional state of the user, providing the user with a personalized server and improving the user's device usage experience.

[0184] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the above device control method is implemented.

[0185] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above device control method is implemented.

[0186] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0187] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0188] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0189] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0190] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0192] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0193] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0194] The above is a detailed introduction to a device control method and apparatus, an electronic device, and a storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a general technician in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A device control method, characterized in that: The method comprises: Acquiring emotion data collected by the first terminal device for the target user; The emotional data is input into a target neural network model, and the emotional state information of the target user is output, wherein the target neural network model is a neural network model that combines a neural network algorithm and an attention mechanism to perform emotional state recognition; One or more second terminal devices are controlled based on the emotional state information.

2. The method according to claim 1, characterized in that The controlling one or more second terminal devices based on the emotional state information includes: Obtaining device status information of one or more second terminal devices; generating a device control strategy for the one or more second terminal devices according to the device state information and the emotional state information; The one or more second terminal devices are controlled according to the device control strategy.

3. The method according to claim 2, characterized in that The generating of the device control strategy of the one or more second terminal devices according to the device state information and the emotional state information includes: Obtain a reinforcement learning algorithm that optimizes and generates device control strategies based on historical device states and historical emotional states; The device state information and the emotional state information are input into the reinforcement learning algorithm to generate an optimized device control strategy.

4. The method according to claim 3, characterized in that Also includes: Obtaining feedback information from the target user regarding the device control strategy; The target neural network model and / or the reinforcement learning algorithm are adjusted based on the feedback information.

5. The method according to claim 3, characterized in that: Also includes: When the environmental state of the second terminal device is updated, the target neural network model and / or the reinforcement learning algorithm are adjusted based on the updated environmental state.

6. The method according to claim 3, characterized in that Also includes: Obtaining feedback information from the target user regarding the device control strategy; Acquiring device usage data of the target user for the one or more second terminal devices; The device linkage strategy is updated based on the feedback information and the device usage data.

7. The method according to any one of claims 1 to 6, characterized in that Before inputting the sentiment data into the target neural network model, the method further includes: The emotional data is preprocessed, wherein the preprocessing includes any one or more of pre-emphasis processing, normalization processing, and endpoint detection.

8. The method according to any one of claims 1, characterized in that The target neural network model is a convolutional neural network model, and the fully connected layer in the convolutional neural network model is connected to the attention layer for executing the attention mechanism. The attention layer is used to receive the feature data output by the fully connected layer, and use the attention mechanism to extract the target emotional features for representing the emotional state of the target user from the model data.

9. The method according to claim 1 or 8, characterized in that: The training process of the target neural network model includes the following steps: determining an initial neural network model from a plurality of pre-trained neural network models; Transfer learning is performed based on the initial neural network model to obtain a target neural network model.

10. A device control device, characterized in that: The device comprises: An emotion data acquisition module, used to acquire emotion data collected by the first terminal device for a target user; An emotional state determination module, used to input the emotional data into a target neural network model and output emotional state information of the target user, wherein the target neural network model is a neural network model that combines a neural network algorithm and an attention mechanism to perform emotional state recognition; A device control module is used to control one or more second terminal devices based on the emotional state information.

11. An electronic device, characterized in that: The device comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the device control method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the device control method according to any one of claims 1 to 9 is implemented.

Images