A smart home appliance user behavior perception and control system

Abstract

The present invention discloses a user behavior perception and control system for smart home appliances, comprising a multimodal sensor module, a data processing and management module, a user behavior perception module, a home appliance scheduling and control module, a system operation monitoring and maintenance module, and a multimodal interaction module. The multimodal sensor module is used to collect data from smart home appliances and user terminals, the data processing and management module is used for preprocessing, integrating and managing sensor data, the user behavior perception module is used to perceive user behavior patterns, the home appliance scheduling and control module is used for adaptive control of smart home appliances, the system operation monitoring and maintenance module is used to monitor the system operation status and maintain it, and the multimodal interaction module is used for multimodal interaction between the system and the user. The present invention discloses a user behavior perception and control system for smart home appliances, and proposes an intelligent perception algorithm based on a neural network to perceive and identify user behavior, and proposes an adaptive control algorithm based on fuzzy reasoning to adaptively control and adjust smart home appliances.

Description

Technical Field

[0001] The present invention relates to the fields of neural networks, fuzzy reasoning and adaptive control, and in particular to a smart home appliance user behavior perception and control system. Background Art

[0002] Neural network technology is a computational model based on the structure of biological nervous systems, designed to solve pattern recognition and learning tasks of complex problems. Neural networks are composed of multiple levels of neurons, each node is interconnected with other nodes, and each connection has a corresponding weight. By adjusting the connection weights, the neural network can learn and adapt to the complex patterns of input data. By training the neural network model, this technology can automatically learn and identify the user's complex behavior patterns, including multimodal behaviors of action, voice, and image. The deep structure and learning ability of the neural network enable it to discover the abstract features hidden behind the data and achieve a deep understanding and prediction of user behavior. By gradually adjusting the weights of the neural network, the system can continuously improve the accuracy of user behavior and provide users with intelligent and stable services.

[0003] Fuzzy reasoning technology is a computational method for processing uncertainty and fuzzy information. It aims to enable the system to process fuzzy and uncertain input information more flexibly by establishing a fuzzy rule base and applying fuzzy reasoning models. Fuzzy reasoning technology simulates the fuzzy reasoning process of humans through fuzzy logic rules and is used to process the fuzziness of user behavior patterns and environmental conditions. Adaptive control technology can dynamically adjust the control strategy of the system during operation according to changes in user behavior to adapt to environmental changes and real-time changes in user needs, and intelligently adjust the control parameters of home appliances to ensure that the system can respond to the real-time needs of users in a timely manner. Through fuzzy reasoning technology and adaptive control technology, the system can more flexibly adjust the working status of home appliances to adapt to the personalized needs of different users, thereby improving the adaptability and flexibility of the user behavior perception and control system of smart home appliances.

[0004] However, the existing smart home appliance user behavior perception and control systems have limitations in the accuracy of user behavior perception and recognition when faced with complex home environments. It is difficult for the system to accurately understand and predict the complex and diverse behaviors of users. At the same time, the system cannot flexibly make intelligent decisions based on the user's real-time needs and different environments, making the system insufficiently flexible in responding to users' personalized needs and different usage scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide a smart home appliance user behavior perception and control system to solve the problem that the existing smart home appliance user behavior perception and control system proposed in the above background technology has limitations in the system's perception and recognition accuracy of user behavior when facing a complex home environment, making it difficult for the system to accurately understand and predict the user's complex and diversified behaviors. At the same time, since the system cannot flexibly make intelligent decisions based on the user's real-time needs and different environments, the system lacks flexibility in responding to the user's personalized needs and different usage scenarios.

[0006] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an intelligent home appliance user behavior perception and control system, comprising a multimodal sensor module, a data processing and management module, a user behavior perception module, a home appliance scheduling and control module, a system operation monitoring and maintenance module, and a multimodal interaction module, characterized in that: the multimodal sensor module is used to integrate multiple sensors, including tactile sensors, sound sensors, image sensors and environmental sensors, to collect various user behavior data in real time, the data processing and management module includes a data preprocessing unit and a data integration and management unit, the data preprocessing unit is used to filter, reduce noise and correct the raw data collected by the sensor to ensure the accuracy and reliability of the data, the data integration and management unit is used to integrate and manage the processed data in real time, including data storage and efficient retrieval functions, and the user behavior perception module proposes an intelligent perception based on a neural network The invention relates to a home appliance scheduling and control module, which comprises a home appliance adaptive control unit and a remote control interface unit. The home appliance adaptive control unit proposes an adaptive control algorithm based on fuzzy reasoning to adapt to the user's behavior pattern and respond to changes in real-time user needs to enhance the flexibility and adaptability of smart home appliances. The remote control interface unit is used to establish a remote communication channel between the user and the system, so that the user can remotely control the smart home appliances and realize remote monitoring and operation of the system by the user. The system operation monitoring and maintenance module is used to monitor the operation status of the user behavior perception and control system of the smart home appliances, detect potential faults and maintain the fault problems to ensure the stable operation of the system. The multimodal interaction module is used to provide a voice interaction interface and a graphic interaction interface to realize efficient interaction between the system and the user.

[0007] Preferably, the multimodal sensor module realizes all-round user behavior data collection by integrating tactile sensors, sound sensors, image sensors and environmental sensors, and ensures the system's efficient and accurate perception of user behavior and environmental changes through comprehensive analysis of the collected multimodal data by subsequent modules.

[0008] Preferably, the data processing and management module includes a data preprocessing unit, which optimizes the raw data collected by the sensor through filtering, noise reduction and correction technology to ensure the accuracy and consistency of the data.

[0009] Preferably, the data processing and management module includes a data integration and management unit, which is responsible for real-time integration and management of processed data, and builds a comprehensive and consistent data set by integrating information that has undergone data preprocessing, ensuring that the system can quickly respond to changes in user behavior, while coordinating and integrating multi-source data to achieve collaborative work among modules within the system.

[0010] Preferably, the user behavior perception module proposes an intelligent perception algorithm based on a neural network, which realizes highly intelligent user behavior perception by comprehensively analyzing and learning multimodal data such as motion, voice, and image, and continuously optimizes the neural network model through iteration to ensure the system's accurate understanding and prediction of user behavior.

[0011] Specifically, the neural network-based intelligent perception algorithm is as follows: first, feature selection is performed on the preprocessed multimodal data set, and the expected values ​​of different user behaviors on specific features are calculated, and the similarities and differences between different user behaviors in the multimodal feature space are judged by calculating the average inter-class distance, and the most informative features for user behavior identification are selected. The specific formula for calculating the average inter-class distance for feature selection is expressed as:

[0012]

[0013] in, It is represented as the average inter-class distance of feature selection, Represented as the index of the feature, Represented as the index of the user behavior type of the tactile and sound modalities after preprocessing and integration operations, Represented as the index of the user behavior type of the image modality after preprocessing and integration operations, Expressed as Type-related user behavior sample data index, Expressed as Type-related user behavior sample data index, Representation and The total amount of user behavior sample data related to the type, Representation and The total amount of user behavior sample data related to the type, and Respectively represent Type and The type of feature selection variance, is represented as the smoothing parameter, and Respectively expressed as Type and Type in Features The expected value on the feature selection is expressed as follows:

[0014]

[0015]

[0016] in, Expressed as Type in Features On The characteristic value of the sample, Expressed as Type in Features On The characteristic value of the sample, It is expressed as a regularization parameter. The regularization term is introduced to consider the skewness of sample distribution, increase the robustness of sample distribution, and improve the generalization and stability of the algorithm for sensor events under different users and environments. Secondly, a feedforward neural network model is constructed, including input layer, hidden layer and output layer. The supervised learning method is adopted, and the error back propagation algorithm is used for training. The network weight is adjusted by gradient descent. The specific formula from the input layer to the hidden layer of the feedforward neural network is expressed as:

[0017]

[0018] in, Represented as a feedforward neural network hidden layer neuron Output information, Represented as the index of the number of neurons in the hidden layer of the feedforward neural network, Represented as the index of the number of neurons in the input layer of the feedforward neural network, It is expressed as the number of neurons in the input layer of the feedforward neural network. Represented as input layer neurons To the hidden layer neurons The connection weights of Represented as input layer neurons Input information, It is expressed as a Sigmoid activation function. The input layer is connected to the hidden layer through weights. After the activation function is used for conversion, the output of the hidden layer is obtained. The specific formula from the hidden layer to the output layer of the feedforward neural network is expressed as:

[0019]

[0020] in, Represented as output layer neurons Output information, Represented as the index of the number of neurons in the output layer of the feedforward neural network, It is expressed as the number of neurons in the hidden layer of the feedforward neural network. Represented as output layer neurons With hidden layer neurons The hidden layer is connected to the output layer through the weight coefficient, and the output of the output layer is converted through the activation function. The difference between the actual output and the target output is measured by the cross entropy loss function to accelerate the convergence of the model. The specific calculation formula of the cross entropy loss function is expressed as:

[0021]

[0022] in, It is expressed as the overall error between the model prediction and the actual situation. It is expressed as the number of neurons in the output layer of the feedforward neural network. Represented as the target output of the feedforward neural network, Represented by a one-hot encoded vector, It is expressed as a natural logarithmic function, and the weights are adjusted by using the cross entropy error and momentum factor to improve the model training efficiency. The specific calculation formula for the feedforward neural network weight adjustment is expressed as:

[0023]

[0024] in, Indicated in The weight coefficient updated in the iteration, represents the learning rate of the feedforward neural network model, represents the momentum factor, Represents the difference between the actual output and the target output. Indicated in The weight coefficient updated in the iteration solves the local minimum problem that occurs in the gradient descent process by using the momentum factor, and increases the memory of the previously updated weights of the model. During the training process, the error is propagated from the output layer to the hidden layer through the back propagation algorithm, and the weight of each layer is adjusted according to the chain rule to improve the accuracy and generalization ability of the feedforward neural network. In summary, the feedforward neural network uses the back propagation mechanism to enable the model to select specific activity feature data from sensor events as input, which can improve the robustness and efficiency of the algorithm and make the feedforward neural network model immune to errors in the training data. In addition, by selecting appropriate feature sub-datasets, the algorithm improves the accuracy of human activity recognition and ensures that the system can perceive user behavior more accurately under different users and environments.

[0025] Preferably, the home appliance scheduling and control module includes a home appliance adaptive control unit, which proposes an adaptive control algorithm based on fuzzy reasoning, and through the construction of a fuzzy reasoning model, intelligently schedules and controls the working status of home appliances according to user behavior patterns to adapt to user behavior patterns and environmental changes, ensuring that the system can be flexibly applied to diverse and complex scenarios and providing a personalized home appliance control experience.

[0026] Specifically, the adaptive control algorithm based on fuzzy reasoning is as follows: First, the multimodal data collected by the sensor is intelligently divided by the K-means clustering method to form different input space regions. The K-means clustering center update formula is specifically expressed as:

[0027]

[0028] in, Expressed as The cluster center is in The clusters after the clustering update, represents the index of the cluster center, Represented as the frequency index of clustering, Represented as clusters belonging to the cluster The amount of data, Represented as a dataset The data points, the K-means clustering distance calculation formula is specifically expressed as:

[0029]

[0030] in, Represented as data points With cluster center The Euclidean distance between Represented as data points and cluster centers The number of dimensions, Expressed as the number of dimensions The index of the input space is divided by the K-means clustering method to avoid the exponential growth of the number of fuzzy rules and improve the interpretability and computational efficiency of the model. Secondly, an adaptive neural fuzzy inference model is constructed based on the multimodal data divided by the K-means clustering method and the output information of the user behavior perception module. The uncertainty of the integrated multimodal data is processed by fuzzy inference, and the fuzzy rules of the adaptive neural fuzzy inference model are specifically expressed as:

[0031]

[0032] in, Represented as the first Fuzzy rules, is represented as the index of the fuzzy rule, Represented as multimodal data of user behavior types in tactile and sound modalities after preprocessing and integration operations, Represented as multimodal data of user behavior types in image modality after preprocessing and integration operations, Represented as a fuzzy set The membership function in Represented as a fuzzy set The membership function in and Represents input and Corresponding to the rules The membership degree of the fuzzy set in Represented as fuzzy rules The output variable, Represented as and rule The relevant weight coefficient, Represented as a rule The output function of the fuzzification process is expressed as follows:

[0033]

[0034]

[0035] in, It is expressed as a parameter that controls the width of the fuzzy set and affects the steepness of the membership function. It is expressed as a parameter that controls the shape of the fuzzy set, affecting the shape of the curves on the left and right sides of the membership function. It is expressed as the parameter controlling the center position of the fuzzy set and determines the center position of the membership function. , , , which makes the membership function flexible and adjustable, so that the shape of the fuzzy set can better adapt to different input data distributions and improve the expressiveness of the adaptive neuro-fuzzy reasoning model. The specific calculation formula of the fuzzy rule weight parameter is expressed as:

[0036]

[0037] in, It is represented as a set of fuzzy rules in the adaptive neural fuzzy reasoning model. The weight coefficients related to each fuzzy rule are normalized to ensure the balance of each rule in the comprehensive contribution. The outputs of all rules are weighted summed to obtain the final output of the model. The output function of the adaptive neural fuzzy reasoning model is expressed as:

[0038]

[0039] in, Represented as fuzzy rules Related adjustment coefficients, finally, through the improved adaptive particle swarm optimization algorithm, the learning process of the adaptive neural fuzzy inference model is accelerated to avoid falling into local optimization and improve the global search ability. The specific calculation formula of the adaptability function is expressed as:

[0040]

[0041] in, Expressed as a fitness function, it is used to measure the output information of the adaptive neuro-fuzzy inference model Actual feedback information after adaptive adjustment with the model The root mean square difference between represents the iteration index of the adaptive neuro-fuzzy inference model, The maximum number of iterations of the adaptive neural fuzzy inference model is shown. The least square method is used to adjust the adaptability function of the adaptive neural fuzzy inference model to improve the learning ability and adaptability of the system. The trade-off between global search and local search in the improved adaptive particle swarm optimization algorithm is balanced by the inertia weight. The inertia weight update formula is specifically expressed as:

[0042]

[0043] in, Indicates The inertia weight in the iteration process controls the movement of particles in the search space. Expressed as the maximum inertia weight, it controls the upper limit of the weight during global search. Represented as the minimum inertia weight, it controls the lower limit of the weight during local search. It is expressed as a nonlinear parameter, which is used to adjust the changing trend of the inertia weight. The speed correction process ensures that the particles have a reasonable moving direction in the search space, including individual experience terms and group collaboration terms, to balance local and global searches and avoid falling into the local optimal solution. The specific calculation formula of speed correction is expressed as:

[0044]

[0045] in, Represented as particles In the The velocity correction term in the iteration process is Represented as particles In the The velocity correction term in the iteration process is Expressed as an inertia coefficient, it is used to adjust the weight of the velocity in the previous iteration in the velocity correction term. It is expressed as an individual learning factor, which is used to control the intensity of the particle adjusting its speed according to individual experience. It is expressed as a group learning factor, which is used to control the intensity of particle speed adjustment based on group collaboration. Represented as the individual optimal position, used to determine the search direction of the particle, Represented as the global optimal position, as the goal of group collaboration, it is used to guide particles to approach the global optimal position. Expressed as The current position information of each particle. In general, the adaptive control algorithm based on fuzzy reasoning has the flexibility to adapt to environmental variables and user behavior patterns. The adaptive neural fuzzy reasoning model constructed through continuous learning and optimization enables the system to adaptively adjust home appliances in a variety of smart home environments, thereby improving the intelligence and adaptability of the user behavior perception and control system of smart home appliances, and providing users with a more intelligent and personalized home experience.

[0046] Preferably, the electrical automation module includes a task identification and switching unit, which ensures that the system can quickly and accurately identify and understand the current electrical automation ecological Internet of Things tasks by utilizing the knowledge information learned by the adaptive meta-learning module, and make accurate decisions for subsequent automation control.

[0047] Preferably, the system operation monitoring and maintenance module monitors the operating status of the system, detects potential faults and abnormal conditions, and performs corresponding maintenance operations to ensure stable operation of the system and improve system reliability and maintenance efficiency.

[0048] Preferably, the multimodal interaction module constructs a voice interaction interface and a graphical user interface and utilizes a multimodal interaction method to ensure that users can choose the most convenient way to interact with the smart home appliance system according to their personal preferences and usage scenarios, thereby improving the overall user experience.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. User behavior perception module proposes an intelligent perception algorithm based on neural network. By using the feedforward neural network model constructed by the back-propagation algorithm, the algorithm can efficiently model the complex relationship between user behavior and environment. In the smart home appliance environment, the feedforward neural network model can understand the user behavior pattern. The algorithm applies error back-propagation in supervised learning, adjusts the network weights by gradient descent, and realizes the training of multi-layer perceptrons, so that the system has a strong learning ability and can explain and adapt to complex sensor events in different environments of smart home appliance systems. By adopting the back-propagation algorithm with momentum factor, the problem of the feedforward neural network model falling into the local minimum can be avoided, and the stability and convergence speed of the system can be improved. In the multimodal data set, the algorithm can be used to train the multi-layer perceptron. In terms of feature selection, the feedforward neural network uses the back-propagation mechanism to enable the model to select specific activity feature data from sensor events as input, which can improve the robustness and efficiency of the algorithm and make the feedforward neural network model immune to errors in the training data. In addition, by selecting appropriate feature sub-datasets, the algorithm improves the accuracy of human activity recognition and ensures that the system can more accurately perceive user behavior under different users and environments, thereby enhancing the reliability and intelligence of the user behavior perception and control system of smart home appliances. In summary, the intelligent perception algorithm based on neural networks helps the user behavior perception and control system of smart home appliances to efficiently perceive user behavior and make accurate behavior predictions, thereby improving the overall intelligence level of the system.

[0051] 2. The adaptive control unit of home appliances proposes an adaptive control algorithm based on fuzzy reasoning. The algorithm realizes intelligent adaptive control of smart homes by analyzing the multimodal data collected by sensors and the user's previous home appliance operation behaviors stored in the system. The intelligent adaptive control of the algorithm can accurately predict and adjust home appliances, making the overall smart home environment more environmentally friendly and comfortable. The algorithm constructs an adaptive neural fuzzy reasoning model and intelligently divides the input space through the K-means clustering method. It can effectively avoid the problem of exponential growth of fuzzy rules and effectively prevent the dimensionality disaster of the adaptive neural fuzzy reasoning model. In terms of control accuracy, the algorithm optimizes the premise and conclusion parameters of the adaptive neural fuzzy reasoning model through an improved adaptive particle swarm optimization mechanism. During the optimization process, the algorithm continuously optimizes a set of premise parameters. The membership function is introduced by reading and adjusting the premise parameters to reduce the root mean square deviation until the closing condition is reached. The improved particle swarm algorithm is efficient in the learning process, can improve the model convergence speed, and effectively avoids local optimization problems. In addition, the algorithm uses the least squares method to adjust the parameters of the adaptive neuro-fuzzy inference model to improve the learning ability and adaptability of the system. In summary, the adaptive control algorithm based on fuzzy reasoning has flexible adaptation to environmental variables and user behavior patterns. The adaptive neuro-fuzzy inference model constructed through continuous learning and optimization enables the system to adaptively adjust home appliances in a variety of smart home environments, improves the intelligence and adaptability of the smart home appliance user behavior perception and control system, and provides users with a more intelligent and personalized home experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The invention is further illustrated by the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0053] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1The present invention provides a smart home appliance user behavior perception and control system, including a multimodal sensor module, a data processing and management module, a user behavior perception module, a home appliance scheduling and control module, a system operation monitoring and maintenance module, and a multimodal interaction module, characterized in that: the multimodal sensor module is used to integrate multiple sensors, including tactile sensors, sound sensors, image sensors and environmental sensors, to collect various user behavior data in real time, the data processing and management module includes a data preprocessing unit and a data integration and management unit, the data preprocessing unit is used to filter, reduce noise and correct the raw data collected by the sensor to ensure the accuracy and reliability of the data, the data integration and management unit is used to integrate and manage the processed data in real time, including data storage and efficient retrieval functions, and the user behavior perception module proposes an intelligent perception algorithm based on a neural network for The complex behavior patterns of users are identified and analyzed to achieve a deep understanding and prediction of user behavior. The home appliance scheduling and control module includes a home appliance adaptive control unit and a remote control interface unit. The home appliance adaptive control unit proposes an adaptive control algorithm based on fuzzy reasoning to adapt to user behavior patterns and respond to changes in real-time user needs to enhance the flexibility and adaptability of smart home appliances. The remote control interface unit is used to establish a remote communication channel between the user and the system, so that the user can remotely control the smart home appliances and realize remote monitoring and operation of the system by the user. The system operation monitoring and maintenance module is used to monitor the operating status of the user behavior perception and control system of smart home appliances, detect potential faults and maintain fault problems to ensure the stable operation of the system. The multimodal interaction module is used to provide voice interaction interface and graphic interaction interface to achieve efficient interaction between the system and the user.

[0056] See also Figure 1 ,Furthermore, the multimodal sensor module realizes all-round user behavior data ,collection by integrating tactile sensors, sound sensors, image sensors and ,environmental sensors. The subsequent modules conduct comprehensive analysis of the ,collected multimodal data to ensure that the system can efficiently and ,accurately perceive user behavior and environmental changes.

[0057] See also Figure 1 ,Furthermore, the data processing and management module includes a data preprocessing unit, which optimizes the raw data collected by the sensor through filtering, noise reduction and correction technology to ensure the accuracy and consistency of the data.

[0058] See also Figure 1,Furthermore, the data processing and management module includes a ,data integration and management unit, which is responsible for real-time ,integration and management of the processed data, and ,constructing a comprehensive and consistent data set by integrating the ,information that has undergone data preprocessing, ensuring that the system can ,quickly respond to changes in user behavior, and ,coordinate and integrate multi-source data to achieve collaborative ,work among the modules within the system.

[0059] See also Figure 1 ,Furthermore, the user behavior perception module proposes an ,intelligent perception algorithm based on neural networks, which ,achieves highly intelligent user behavior perception by ,comprehensively analyzing and learning multi-modal data of motion, voice and ,image, and continuously optimizes the neural network model through iteration ,to ensure the system ’s accurate understanding and prediction of user ,behavior.

[0060] See also Figure 1 ,Furthermore, the intelligent perception algorithm based on neural network is as follows: ,First, feature selection is performed on the preprocessed multimodal data set. ,The expected values ​​of different user behaviors on specific features are calculated ,and the average inter-class distance is calculated to determine the similarities and differences between different ,user behaviors in the multimodal feature space, and the most informative features for user behavior ,recognition are selected. ,The specific formula for calculating the average inter-class distance for feature selection is expressed as:

[0061]

[0062] in, It is represented as the average inter-class distance of feature selection, Represented as the index of the feature, Represented as the index of the user behavior type of the tactile and sound modalities after preprocessing and integration operations, Represented as the index of the user behavior type of the image modality after preprocessing and integration operations, Expressed as Type-related user behavior sample data index, Expressed as Type-related user behavior sample data index, Representation and The total amount of user behavior sample data related to the type, Representation and The total amount of user behavior sample data related to the type, and Respectively represent Type and The type of feature selection variance, is represented as the smoothing parameter, and Respectively expressed as Type and Type in Features The expected value on the feature selection is expressed as follows:

[0063]

[0064]

[0065] in, Expressed as Type in Features On The characteristic value of the sample, Expressed as Type in Features On The characteristic value of the sample, It is expressed as a regularization parameter. The regularization term is introduced to consider the skewness of sample distribution, increase the robustness of sample distribution, and improve the generalization and stability of the algorithm for sensor events under different users and environments. Secondly, a feedforward neural network model is constructed, including input layer, hidden layer and output layer. The supervised learning method is adopted, and the error back propagation algorithm is used for training. The network weight is adjusted by gradient descent. The specific formula from the input layer to the hidden layer of the feedforward neural network is expressed as:

[0066]

[0067] in, Represented as a feedforward neural network hidden layer neuron Output information, Represented as the index of the number of neurons in the hidden layer of the feedforward neural network, Represented as the index of the number of neurons in the input layer of the feedforward neural network, It is expressed as the number of neurons in the input layer of the feedforward neural network. Represented as input layer neurons To the hidden layer neurons The connection weights, Represented as input layer neurons Input information, It is expressed as a Sigmoid activation function. The input layer is connected to the hidden layer through weights. After the activation function is used for conversion, the output of the hidden layer is obtained. The specific formula from the hidden layer to the output layer of the feedforward neural network is expressed as:

[0068]

[0069] in, Represented as output layer neurons Output information, Represented as the index of the number of neurons in the output layer of the feedforward neural network, It is expressed as the number of neurons in the hidden layer of the feedforward neural network. Represented as output layer neurons With hidden layer neurons The hidden layer is connected to the output layer through the weight coefficient, and the output of the output layer is converted through the activation function. The difference between the actual output and the target output is measured by the cross entropy loss function to accelerate the convergence of the model. The specific calculation formula of the cross entropy loss function is expressed as:

[0070]

[0071] in, It is expressed as the overall error between the model prediction and the actual situation. It is expressed as the number of neurons in the output layer of the feedforward neural network. Represented as the target output of the feedforward neural network, Represented by a one-hot encoded vector, It is expressed as a natural logarithmic function, and the weights are adjusted by using the cross entropy error and momentum factor to improve the model training efficiency. The specific calculation formula for the feedforward neural network weight adjustment is expressed as:

[0072]

[0073] in, Indicated in The weight coefficient updated in the iteration, represents the learning rate of the feedforward neural network model, represents the momentum factor, Represents the difference between the actual output and the target output. Indicated in The weight coefficient updated in the iteration solves the local minimum problem that occurs in the gradient descent process by using the momentum factor, and increases the memory of the previously updated weights of the model. During the training process, the error is propagated from the output layer to the hidden layer through the back propagation algorithm, and the weight of each layer is adjusted according to the chain rule to improve the accuracy and generalization ability of the feedforward neural network. In summary, the feedforward neural network uses the back propagation mechanism to enable the model to select specific activity feature data from sensor events as input, which can improve the robustness and efficiency of the algorithm and make the feedforward neural network model immune to errors in the training data. In addition, by selecting appropriate feature sub-datasets, the algorithm improves the accuracy of human activity recognition and ensures that the system can perceive user behavior more accurately under different users and environments.

[0074] See also Figure 1Furthermore, the home appliance scheduling and control module includes a home appliance adaptive control unit, which proposes an adaptive control algorithm based on fuzzy reasoning. By constructing a fuzzy reasoning model, it intelligently schedules and controls the working status of home appliances according to user behavior patterns to adapt to user behavior patterns and environmental changes, ensuring that the system can be flexibly applied to diverse and complex scenarios and providing a personalized home appliance control experience.

[0075] See also Figure 1 ,Furthermore, the adaptive control algorithm based on fuzzy reasoning is as follows: ,First, the multimodal data collected by the sensor is intelligently divided by ,the K-means clustering method to form different input space regions. ,The K-means clustering center update formula is specifically expressed as:

[0076]

[0077] in, Expressed as The cluster center is in The clusters after the clustering update, represents the index of the cluster center, Represented as the frequency index of clustering, Represented as clusters belonging to the cluster The amount of data, Represented as a dataset The data points, the K-means clustering distance calculation formula is specifically expressed as:

[0078]

[0079] in, Represented as data points With cluster center The Euclidean distance between Represented as data points and cluster centers The number of dimensions, Expressed as the number of dimensions The index of the input space is divided by the K-means clustering method to avoid the exponential growth of the number of fuzzy rules and improve the interpretability and computational efficiency of the model. Secondly, an adaptive neural fuzzy inference model is constructed based on the multimodal data divided by the K-means clustering method and the output information of the user behavior perception module. The uncertainty of the integrated multimodal data is processed by fuzzy inference, and the fuzzy rules of the adaptive neural fuzzy inference model are specifically expressed as:

[0080]

[0081] in, Represented as the first Fuzzy rules, is represented as the index of the fuzzy rule, Represented as multimodal data of user behavior types in tactile and sound modalities after preprocessing and integration operations, Represented as multimodal data of user behavior types in image modality after preprocessing and integration operations, Represented as a fuzzy set The membership function in Represented as a fuzzy set The membership function in and Represents input and Corresponding to the rules The membership degree of the fuzzy set in Represented as fuzzy rules The output variable, Represented as and rule The relevant weight coefficient, Represented as a rule The output function of the fuzzification process is expressed as follows:

[0082]

[0083]

[0084] in, It is expressed as a parameter that controls the width of the fuzzy set and affects the steepness of the membership function. It is expressed as a parameter that controls the shape of the fuzzy set, affecting the shape of the curves on the left and right sides of the membership function. It is expressed as the parameter controlling the center position of the fuzzy set and determines the center position of the membership function. , , , which makes the membership function flexible and adjustable, so that the shape of the fuzzy set can better adapt to different input data distributions and improve the expressiveness of the adaptive neuro-fuzzy reasoning model. The specific calculation formula of the fuzzy rule weight parameter is expressed as:

[0085]

[0086] in, It is represented as a set of fuzzy rules in the adaptive neural fuzzy reasoning model. The weight coefficients related to each fuzzy rule are normalized to ensure the balance of each rule in the comprehensive contribution. The outputs of all rules are weighted summed to obtain the final output of the model. The output function of the adaptive neural fuzzy reasoning model is expressed as:

[0087]

[0088] in, Represented as fuzzy rules Related adjustment coefficients, finally, through the improved adaptive particle swarm optimization algorithm, the learning process of the adaptive neural fuzzy inference model is accelerated to avoid falling into local optimization and improve the global search ability. The specific calculation formula of the adaptability function is expressed as:

[0089]

[0090] in, Expressed as a fitness function, it is used to measure the output information of the adaptive neuro-fuzzy inference model Actual feedback information after adaptive adjustment with the model The root mean square difference between represents the iteration index of the adaptive neuro-fuzzy inference model, The maximum number of iterations of the adaptive neural fuzzy inference model is shown. The least square method is used to adjust the adaptability function of the adaptive neural fuzzy inference model to improve the learning ability and adaptability of the system. The trade-off between global search and local search in the improved adaptive particle swarm optimization algorithm is balanced by the inertia weight. The inertia weight update formula is specifically expressed as:

[0091]

[0092] in, Indicates The inertia weight in the iteration process controls the movement of particles in the search space. Expressed as the maximum inertia weight, it controls the upper limit of the weight during global search. Represented as the minimum inertia weight, it controls the lower limit of the weight during local search. It is expressed as a nonlinear parameter, which is used to adjust the changing trend of the inertia weight. The speed correction process ensures that the particles have a reasonable moving direction in the search space, including individual experience terms and group collaboration terms, to balance local and global searches and avoid falling into the local optimal solution. The specific calculation formula of speed correction is expressed as:

[0093]

[0094] in, Represented as particles In the The velocity correction term in the iteration process is Represented as particles In the The velocity correction term in the iteration process is Expressed as an inertia coefficient, it is used to adjust the weight of the velocity in the previous iteration in the velocity correction term. It is expressed as an individual learning factor, which is used to control the intensity of the particle adjusting its speed according to individual experience. It is expressed as a group learning factor, which is used to control the intensity of particle speed adjustment based on group collaboration. Represented as the individual optimal position, used to determine the search direction of the particle, Represented as the global optimal position, as the goal of group collaboration, it is used to guide particles to approach the global optimal position. Expressed as The current position information of each particle. In general, the adaptive control algorithm based on fuzzy reasoning has the flexibility to adapt to environmental variables and user behavior patterns. The adaptive neural fuzzy reasoning model constructed through continuous learning and optimization enables the system to adaptively adjust home appliances in a variety of smart home environments, thereby improving the intelligence and adaptability of the user behavior perception and control system of smart home appliances, and providing users with a more intelligent and personalized home experience.

[0095] See also Figure 1 ,Furthermore, the system operation monitoring and maintenance module monitors the ,operating status of the system, detects potential faults and abnormal ,situations, and performs corresponding maintenance operations to ensure ,the stable operation of the system and improve the reliability and ,maintenance efficiency of the system.

[0096] See also Figure 1 ,Furthermore, the multimodal interaction module builds a voice interaction interface and ,graphical user interface and utilizes multimodal interaction methods to ensure ,that users can choose the most convenient way to interact with the smart ,home appliance system according to their personal preferences and ,usage scenarios, thereby improving the overall user experience.

[0097] When used specifically, first, the multimodal sensor module integrates multiple sensors, including tactile sensors, sound sensors, image sensors and environmental sensors, to collect various user behavior data in real time. Then, the data processing and management module includes a data preprocessing unit and a data integration and management unit. The data preprocessing unit filters, reduces noise and corrects the original data collected by the sensor to ensure the accuracy and reliability of the data. The data integration and management unit integrates and manages the processed data in real time, including data storage and efficient retrieval functions. Subsequently, the user behavior perception module proposes an intelligent perception algorithm based on neural networks to identify and analyze the user's complex behavior patterns, thereby achieving a deep understanding and prediction of user behavior. The home appliance scheduling and control module includes It includes the home appliance adaptive control unit and the remote control interface unit. The home appliance adaptive control unit proposes an adaptive control algorithm based on fuzzy reasoning to adapt to the user's behavior pattern and respond to changes in real-time user needs to improve the flexibility and adaptability of smart home appliances. Secondly, the remote control interface unit establishes a remote communication channel between the user and the system, allowing the user to remotely control the smart home appliances and realize the user's remote monitoring and operation of the system. Finally, the system operation monitoring and maintenance module detects the user behavior perception system of smart home appliances, controls the operating status of the system, detects potential faults and maintains fault problems to ensure the stable operation of the system. The multimodal interaction module provides voice interaction interface and graphic interaction interface to achieve efficient interaction between the system and the user.

[0098] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A smart home appliance user behavior perception and control system, comprising a multimodal sensor module, a data processing and management module, a user behavior perception module, a home appliance scheduling and control module, a system operation monitoring and maintenance module, and a multimodal interaction module, characterized in that: The multimodal sensor module is used to integrate multiple sensors, including tactile sensors, sound sensors, image sensors and environmental sensors, to collect various user behavior data in real time. The data processing and management module includes a data preprocessing unit and a data integration and management unit. The data preprocessing unit is used to filter, reduce noise and correct the original data collected by the sensor to ensure the accuracy and reliability of the data. The data integration and management unit is used to integrate and manage the processed data in real time, including data storage and efficient retrieval functions. The user behavior perception module proposes an intelligent perception algorithm based on a neural network to identify and analyze the user's composite behavior pattern and achieve a deep understanding and prediction of the user's behavior. The home appliance scheduling and control module includes The system comprises a home appliance adaptive control unit and a remote control interface unit. The home appliance adaptive control unit proposes an adaptive control algorithm based on fuzzy reasoning to adapt to the user's behavior pattern and respond to changes in real-time user needs to improve the flexibility and adaptability of smart home appliances. The remote control interface unit is used to establish a remote communication channel between the user and the system, so that the user can remotely control the smart home appliance and realize the user's remote monitoring and operation of the system. The system operation monitoring and maintenance module is used to monitor the operation status of the smart home appliance user behavior perception and control system, detect potential faults and maintain fault problems to ensure the stable operation of the system. The multimodal interaction module is used to provide a voice interaction interface and a graphic interaction interface to realize efficient interaction between the system and the user. The user behavior perception module proposes an intelligent perception algorithm based on a neural network, which realizes highly intelligent user behavior perception by comprehensively analyzing and learning multimodal data such as action, voice, and image, and continuously optimizes the neural network model through iteration to ensure that the system accurately understands and predicts user behavior; First, feature selection is performed on the preprocessed multimodal data set. The specific formula for calculating the average inter-class distance of feature selection is expressed as: Where D(τ) represents the average inter-class distance of feature selection, τ represents the index of the feature, x represents the index of the user behavior type of the tactile and sound modalities after preprocessing and integration, y represents the index of the user behavior type of the image modality after preprocessing and integration, i represents the index of the user behavior sample data related to the x type, j represents the index of the user behavior sample data related to the y type, and N x N represents the total amount of user behavior sample data related to type x, y represents the total amount of user behavior sample data related to type y, ξ 2 (x i ,τ) and ξ 2 (y j ,τ) represent the variance of feature selection related to type x and type y, ∈ represents the smoothing parameter, E(x,τ) and E(y,τ) represent the expected values ​​of type x and type y on feature τ, respectively. A feedforward neural network model is constructed, including input layer, hidden layer and output layer. The supervised learning method is adopted, and the error back propagation algorithm is used for training. The network weights are adjusted by gradient descent. The specific formula from the input layer to the hidden layer of the feedforward neural network is expressed as: Among them, IN h It represents the output information of the hidden layer neuron h of the feedforward neural network, h represents the index of the number of neurons in the hidden layer of the feedforward neural network, l represents the index of the number of neurons in the input layer of the feedforward neural network, n represents the number of neurons in the input layer of the feedforward neural network, and v lh It is represented as the connection weight from the input layer neuron l to the hidden layer neuron h, x l It is represented as the input information of the input layer neuron l, f(·) is represented as the Sigmoid activation function, and the specific formula from the hidden layer to the output layer of the feedforward neural network is expressed as: Among them, OUT g It represents the output information of the output layer neuron g, g represents the index of the number of neurons in the output layer of the feedforward neural network, m represents the number of neurons in the hidden layer of the feedforward neural network, and w gh It is expressed as the weight coefficient between the output layer neuron g and the hidden layer neuron h. The specific calculation formula for the weight adjustment of the feedforward neural network is expressed as: Among them, Δw gh (r+1) represents the weight coefficient updated in the r+1 iteration, η represents the learning rate of the feedforward neural network model, FAC represents the momentum factor, Represents the difference between the actual output and the target output, Δw gh (r) represents the weight coefficient updated in r iterations.

2. The smart home appliance user behavior perception and control system according to claim 1, characterized in that: The multimodal sensor module integrates tactile sensors, sound sensors, image sensors and environmental sensors to achieve all-round user behavior data collection. The subsequent modules conduct comprehensive analysis of the collected multimodal data to ensure that the system can perceive user behavior and environmental changes efficiently and accurately.

3. The smart home appliance user behavior perception and control system according to claim 1, characterized in that: The data processing and management module includes a data preprocessing unit, which optimizes the raw data collected by the sensor through filtering, noise reduction and correction technology to ensure the accuracy and consistency of the data.

4. The smart home appliance user behavior perception and control system according to claim 1, characterized in that: The data processing and management module includes a data integration and management unit, which is responsible for real-time integration and management of processed data. By integrating the information that has undergone data preprocessing, a comprehensive and consistent data set is constructed to ensure that the system can quickly respond to changes in user behavior. At the same time, multi-source data is coordinated and integrated to achieve collaborative work among the modules within the system.

5. The smart home appliance user behavior perception and control system according to claim 1, characterized in that: The home appliance scheduling and control module includes a home appliance adaptive control unit. The home appliance adaptive control unit proposes an adaptive control algorithm based on fuzzy reasoning. By constructing a fuzzy reasoning model, the working status of home appliances is intelligently scheduled and controlled according to user behavior patterns to adapt to user behavior patterns and environmental changes, ensuring that the system can be flexibly applied to diverse and complex scenarios and providing a personalized home appliance control experience.

6. The smart home appliance user behavior perception and control system according to claim 5, characterized in that: The fuzzy rules for constructing the adaptive neuro-fuzzy inference model are specifically expressed as: Among them, Rule i represents the i-th fuzzy rule in the adaptive neural-fuzzy inference model, i represents the index of the fuzzy rule, x represents the index of the user behavior type of the tactile and sound modalities after the preprocessing integration operation, y represents the index of the user behavior type of the image modality after the preprocessing integration operation, Represented as fuzzy set A i The membership function in It is represented as fuzzy set B i The membership function in and They represent the membership of the fuzzy set in rule i for input x and y, respectively, and z i is represented as the output variable of fuzzy rule i, ω i Expressed as the weight coefficient associated with rule i, f i (x, y) is expressed as the output function of rule i, and the specific calculation formula of the fuzzy rule weight parameter is expressed as: Among them, I represents the set of fuzzy rules in the adaptive neural fuzzy reasoning model. The weight coefficients related to each fuzzy rule are normalized to ensure the balance of each rule in the comprehensive contribution. The outputs of all rules are weighted summed to obtain the final output of the model. The output function of the adaptive neural fuzzy reasoning model is expressed as: Among them, α i It is expressed as the adjustment coefficient related to fuzzy rule i. Finally, the learning process of the adaptive neural fuzzy inference model is accelerated by the improved adaptive particle swarm optimization algorithm to avoid falling into local optimization and improve the global search capability. The specific calculation formula of the adaptability function is expressed as: Among them, F represents the adaptability function, which is used to measure the output information O of the adaptive neuro-fuzzy inference model and the actual feedback information O after the model is adaptively adjusted. * , r represents the iteration index of the adaptive neural fuzzy inference model, and R represents the maximum iteration number of the adaptive neural fuzzy inference model.

7. The smart home appliance user behavior perception and control system according to claim 1, characterized in that: The home appliance scheduling and control module includes a remote control interface unit, which realizes the user's remote control of smart home appliances by building a remote communication channel. Through a safe and reliable communication protocol, it ensures that the user can remotely control home appliances efficiently, thereby improving the convenience and flexibility of the system.

8. The smart home appliance user behavior perception and control system according to claim 1, characterized in that: The system operation monitoring and maintenance module monitors the operating status of the system, detects potential faults and abnormal conditions, and performs corresponding maintenance operations to ensure stable operation of the system and improve system reliability and maintenance efficiency.

Images