A smart home remote control early warning method and system

Abstract

The present invention proposes a method and system for early warning of remote control of smart home. The method includes: obtaining multimodal data of operation of smart home appliances in smart home; cleaning and normalizing the multimodal data, extracting features of each modality after cleaning and normalization, and forming a comprehensive feature vector with the features extracted from each modality; inputting the comprehensive feature vector into a machine learning model for prediction, and obtaining a predicted operation state of the home appliance, and issuing an early warning when the predicted operation state of the home appliance exceeds a threshold; after the early warning, combining historical maintenance records and service life to generate maintenance suggestions. Through this method and the corresponding system, real-time monitoring and fault early warning of smart home devices can be achieved.

Description

Technical Field

[0001] The present invention provides a smart home remote control and early warning method and system, and relates to the technical field of smart home systems. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology and artificial intelligence (AI), smart home systems have become an important part of modern family life. Smart home systems enable remote monitoring and control of home appliances through various sensors and networked devices. These systems not only improve the convenience of life, but also play an important role in energy saving, security and other aspects. However, how to effectively monitor the operating status of home appliances and issue early warnings before potential failures occur is still a problem that needs to be solved.

[0003] Traditional methods of monitoring home appliances mainly rely on regular maintenance and manual inspections, which are not only time-consuming and labor-intensive, but also fail to detect potential faults in a timely manner. As the complexity and diversity of home appliances increase, the limitations of traditional methods become more and more obvious. Therefore, smart home remote control and early warning systems based on the Internet of Things and artificial intelligence technologies have emerged. Summary of the invention

[0004] The present invention provides a smart home remote control early warning method and system to solve the above-mentioned problems:

[0005] The present invention provides a smart home remote control early warning method, the method comprising:

[0006] Obtain multimodal data on the operation of smart appliances in smart homes;

[0007] The multimodal data is cleaned and normalized, and after the cleaning and normalization, features of the data of each modality are extracted, and the features extracted from each modality are combined into a comprehensive feature vector;

[0008] Inputting the comprehensive feature vector into a machine learning model for prediction, and issuing an early warning when the comprehensive feature vector exceeds a threshold;

[0009] After the early warning, maintenance recommendations are generated based on historical maintenance records and service life.

[0010] Furthermore, multimodal data of the operation of smart appliances in smart homes is obtained, including:

[0011] A current sensor disposed at an input end of a power line of the smart home appliance detects a current passing through the smart home appliance;

[0012] A voltage sensor disposed at a power input terminal of the smart home appliance detects a voltage passing through the smart home appliance;

[0013] A power sensor disposed at the power input terminal of the smart home appliance detects the power passing through the smart home appliance;

[0014] A temperature sensor disposed at a heating element of the smart home appliance detects the temperature of the smart home appliance;

[0015] A vibration frequency sensor disposed at a component of the smart home appliance that generates vibration detects the vibration frequency of the smart home appliance;

[0016] The multimodal data is collected according to the collection frequency. Specifically, the collection frequency model is:

[0017]

[0018] in, is the minimum acquisition frequency, is the maximum acquisition frequency, is the minimum value of the rate of change, is the maximum value of the rate of change, It is the rate of change of the data collected at time t relative to the previous time.

[0019] Furthermore, the multimodal data is cleaned and normalized, and after the cleaning and normalization, feature extraction is performed on the data of each modality, and the features extracted from each modality are combined into a comprehensive feature vector, including:

[0020] The multimodal data is cleaned and normalized, wherein the cleaning includes: processing missing values ​​and outliers, processing missing values ​​means filling missing values ​​with the previous valid values ​​collected by the sensor, processing outliers means determining whether the data has outliers with a preset first threshold, and if there are outliers, replacing them with the mean of the collected data, and the normalization process

[0021]

[0022] in, and are the mean and standard deviation of the data of sensor i, is the data collected by sensor i at time t, is the normalized data collected by sensor i at time t;

[0023] After cleaning and normalization, the following model is used to extract features from the data of each modality:

[0024]

[0025] in, represents the data of sensor i in the frequency domain, ω is the angular frequency, and represents the variables of different frequency components. is the normalized data collected by sensor i at time t, t represents the time variable, and j is a complex unit;

[0026]

[0027] in, represents the characteristics of the data of sensor i in the time-scale domain, a represents the scale parameter, b represents the displacement parameter, t represents the time variable, represents the mother wavelet function;

[0028] Combine the features extracted from each mode above into a comprehensive feature vector.

[0029]

[0030] in, represents the comprehensive feature vector of each sensor at time t, Represents the data of the current sensor in the frequency domain, Represents the data of the voltage sensor in the frequency domain, Represents the data of the power sensor in the frequency domain, Represents the characteristics of the current sensor data in the time-scale domain, Represents the characteristics of the voltage sensor data in the time-scale domain, Represents the characteristics of the power sensor data in the time-scale domain.

[0031] Further, the comprehensive feature vector is input into a machine learning model for prediction, including:

[0032] The integrated feature vectors fused from the historical data are divided into a training set and a validation set. The training set is used to train the model by iteratively optimizing the parameters. The model performance is evaluated on the validation set, and the model parameters are adjusted to obtain a trained prediction model.

[0033] The newly fused comprehensive feature vector is input into the prediction model to generate a prediction result.

[0034]

[0035] in, is the predicted operating status of the home appliance. represents the comprehensive feature vector of each sensor at time t, represents the parameters of the prediction model, and f represents the prediction function;

[0036] When the prediction result is greater than a first preset threshold, an early warning message is sent to the user to remind the user of a potential fault.

[0037] Furthermore, after the warning, maintenance suggestions are generated based on historical maintenance records and service life, including:

[0038] when When the user is prompted to replace the content, a suggestion for immediate replacement is generated and sent to the user;

[0039] when When the repair is required, a suggestion is generated and sent to the user;

[0040] when When the system is running, it generates suggestions for normal operation and sends them to the user;

[0041] in, , represents the comprehensive score of historical maintenance records, n represents the number of maintenance times, is represented as the score of the ith maintenance, is the time of the ith maintenance, λ is the attenuation coefficient, indicating that the impact of historical maintenance gradually weakens over time; A= , T1 is the usage time of the smart appliance, the usage time of the smart appliance is the time interval between the current time and the installation time, T2 is the expected life span of the smart appliance, m1 is the preset second threshold, m2 is the preset third threshold, α, β and λ are weight coefficients.

[0042] The present invention proposes a smart home remote control and early warning system, the system comprising:

[0043] A data acquisition module is used to acquire multimodal data of the operation of smart appliances in a smart home;

[0044] A comprehensive feature vector generation module is used to clean and normalize the multimodal data, extract features from the data of each modality after cleaning and normalization, and combine the features extracted from each modality into a comprehensive feature vector;

[0045] An early warning module, used for inputting the comprehensive feature vector into a machine learning model for prediction, and issuing an early warning when the comprehensive feature vector exceeds a threshold;

[0046] The maintenance suggestion generation module is used to generate maintenance suggestions based on historical maintenance records and service life after early warning.

[0047] Furthermore, the data acquisition module includes:

[0048] A current data acquisition module, wherein a current sensor disposed at the input end of the power line of the smart home appliance detects the current passing through the smart home appliance;

[0049] A voltage data acquisition module, wherein a voltage sensor disposed at the power input terminal of the smart home appliance detects the voltage passing through the smart home appliance;

[0050] A power data acquisition module, wherein a power sensor disposed at the power input terminal of the smart home appliance detects the power passing through the smart home appliance;

[0051] A temperature data acquisition module, wherein a temperature sensor disposed at a heating element of the smart home appliance detects the temperature of the smart home appliance;

[0052] A vibration frequency data acquisition module, wherein a vibration frequency sensor disposed at a component of the smart home appliance that generates vibration detects the vibration frequency of the smart home appliance;

[0053] The acquisition frequency calculation module collects the multimodal data according to the acquisition frequency. Specifically, the acquisition frequency model is:

[0054]

[0055] in, is the minimum acquisition frequency, is the maximum acquisition frequency, is the minimum value of the rate of change, is the maximum value of the rate of change, It is the rate of change of the data collected at time t relative to the previous time.

[0056] Furthermore, the module for generating a comprehensive feature vector includes:

[0057] The preprocessing module is used to clean and normalize the multimodal data. The cleaning includes: processing missing values ​​and outliers. Processing missing values ​​is to fill the missing values ​​with the previous valid values ​​collected by the sensor. Processing outliers is to determine whether the data has outliers with a preset first threshold. If there are outliers, replace them with the mean of the collected data. The normalization processing model is:

[0058]

[0059] in, and are the mean and standard deviation of the data of sensor i, is the data collected by sensor i at time t, is the normalized data collected by sensor i at time t;

[0060] The feature extraction module is used to extract features from the data of each modality using the following model after cleaning and normalization.

[0061]

[0062] in, represents the data of sensor i in the frequency domain, ω is the angular frequency, and represents the variables of different frequency components. is the normalized data collected by sensor i at time t, t represents the time variable, and j is a complex unit;

[0063]

[0064] in, represents the characteristics of the data of sensor i in the time-scale domain, a represents the scale parameter, b represents the displacement parameter, t represents the time variable, represents the mother wavelet function;

[0065] The combined feature vector module is used to combine the features extracted from each mode into a comprehensive feature vector.

[0066]

[0067] in, represents the comprehensive feature vector of each sensor at time t, Represents the data of the current sensor in the frequency domain, Represents the data of the voltage sensor in the frequency domain, Represents the data of the power sensor in the frequency domain, Represents the characteristics of the current sensor data in the time-scale domain, Represents the characteristics of the voltage sensor data in the time-scale domain, Represents the characteristics of the power sensor data in the time-scale domain.

[0068] Furthermore, the early warning module includes:

[0069] The training model module is used to divide the integrated feature vectors fused in the historical data into a training set and a validation set, use the training set to train the model by iteratively optimizing the parameters, evaluate the model performance on the validation set, and adjust the model parameters to obtain a trained prediction model;

[0070] A prediction result generating module is used to input the newly fused comprehensive feature vector into the prediction model to generate a prediction result.

[0071]

[0072] in, is the predicted operating status of the home appliance. represents the comprehensive feature vector of each sensor at time t, represents the parameters of the prediction model, and f represents the prediction function;

[0073] The module for sending early warning information is used to send early warning information to the user to remind the user of potential failure when the prediction result is greater than a first preset threshold.

[0074] Furthermore, the maintenance suggestion generating module includes:

[0075] Replace the module immediately for When the user is prompted to replace the content, a suggestion for immediate replacement is generated and sent to the user;

[0076] Repair module for When the repair is required, a suggestion is generated and sent to the user;

[0077] Normal operation module, used when When the system is running, it generates suggestions for normal operation and sends them to the user;

[0078] in, , represents the comprehensive score of historical maintenance records, n represents the number of maintenance times, is represented as the score of the ith maintenance, is the time of the ith maintenance, λ is the attenuation coefficient, indicating that the impact of historical maintenance gradually weakens over time; A= , T1 is the usage time of the smart appliance, the usage time of the smart appliance is the time interval between the current time and the installation time, T2 is the expected life span of the smart appliance, m1 is the preset second threshold, m2 is the preset third threshold, α, β and λ are weight coefficients.

[0079] The beneficial effects of the present invention are as follows: the operating status of smart home devices can be monitored in real time, and the health status of the devices can be fully understood through comprehensive analysis of multimodal data; fault warning, through the prediction of machine learning models, can give early warning before the fault occurs, thereby reducing the loss and inconvenience caused by sudden equipment failure; data-driven, using multimodal data and machine learning technology, the warning system has high accuracy and robustness, can effectively identify potential faults, and improve the system's warning ability; maintenance suggestions, combined with historical maintenance records and years of use, generate accurate maintenance suggestions to help users maintain or replace equipment in a timely manner, extend the service life of the equipment, and improve user experience; the smart home system reduces the need for manual inspection and maintenance through automated monitoring and warning, improves management efficiency, saves time and labor costs; through intelligent warning and maintenance suggestions, users can use smart home devices more safely and improve the overall quality of life; the technical solution realizes real-time monitoring and fault warning of smart home devices through intelligent data monitoring and analysis, improves the security, reliability and user experience of the system, and has significant application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of a smart home remote control and early warning method according to the present invention. DETAILED DESCRIPTION

[0081] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0082] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. The embodiments described are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0084] One embodiment of the present invention provides a smart home remote control early warning method, the method comprising:

[0085] Obtain multimodal data on the operation of smart appliances in smart homes;

[0086] The multimodal data is cleaned and normalized, and after the cleaning and normalization, features of the data of each modality are extracted, and the features extracted from each modality are combined into a comprehensive feature vector;

[0087] Inputting the comprehensive feature vector into a machine learning model for prediction, and issuing an early warning when the comprehensive feature vector exceeds a threshold;

[0088] After the early warning, maintenance recommendations are generated based on historical maintenance records and service life.

[0089] The working principle and effect of the above technical solution are as follows: multimodal data acquisition, smart home devices collect data through a variety of sensors, including but not limited to current, voltage, power, temperature, vibration frequency and other related environmental parameters, etc. These sensor data constitute multimodal data, which can fully reflect the operating status of home appliances; data cleaning and normalization,

[0090] The collected multimodal data is cleaned to remove noise and outliers to ensure the integrity and accuracy of the data; the normalization process standardizes the data so that the data of different modes have the same scale, which is convenient for subsequent analysis and processing; the cleaned and normalized multimodal data is subjected to feature extraction, and useful features are extracted from the data of each mode. The features extracted from each mode are combined into a comprehensive feature vector (including the features of all sensor data) for more comprehensive analysis; the comprehensive feature vector is input into the trained machine learning model for prediction. Commonly used models include long short-term memory network (LSTM), support vector machine (SVM), etc.; the model is used to predict the operating status of household appliances. When the prediction result of the comprehensive feature vector exceeds the set threshold, the system will issue a warning signal; the warning information is sent to the user through the smart home system to remind the user to pay attention to potential faults. According to the warning results, combined with the historical maintenance records and service life of the household appliances, maintenance recommendations are generated. It can monitor the operating status of smart home devices in real time, and comprehensively understand the health status of the equipment through comprehensive analysis of multimodal data; fault warning, through the prediction of machine learning models, it can give early warning before the fault occurs, reducing the loss and inconvenience caused by sudden equipment failure; data-driven, using multimodal data and machine learning technology, the warning system has high accuracy and robustness, can effectively identify potential faults, and improve the system's warning ability; maintenance suggestions, combined with historical maintenance records and years of use, generate accurate maintenance suggestions to help users maintain or replace equipment in time, extend the service life of equipment, and improve user experience; through automated monitoring and early warning, the smart home system reduces the need for manual inspection and maintenance, improves management efficiency, saves time and labor costs; through intelligent early warning and maintenance suggestions, users can use smart home devices more safely and improve the overall quality of life; this technical solution realizes real-time monitoring and fault warning of smart home devices through intelligent data monitoring and analysis, improves the security, reliability and user experience of the system, and has significant application value and promotion prospects.

[0091] In one embodiment of the present invention, obtaining multimodal data of operation of smart appliances in a smart home includes:

[0092] A current sensor disposed at an input end of a power line of the smart home appliance detects a current passing through the smart home appliance;

[0093] A voltage sensor disposed at a power input terminal of the smart home appliance detects a voltage passing through the smart home appliance;

[0094] A power sensor disposed at the power input terminal of the smart home appliance detects the power passing through the smart home appliance;

[0095] A temperature sensor disposed at a heating element of the smart home appliance detects the temperature of the smart home appliance;

[0096] A vibration frequency sensor disposed at a component of the smart home appliance that generates vibration detects the vibration frequency of the smart home appliance;

[0097] The multimodal data is collected according to the collection frequency. Specifically, the collection frequency model is:

[0098]

[0099] in, is the minimum acquisition frequency, is the maximum acquisition frequency, is the minimum value of the rate of change, is the maximum value of the rate of change, It is the rate of change of the data collected at time t relative to the previous time.

[0100] The working principle and effect of the above technical solution are as follows: by setting current sensors, voltage sensors, power sensors, temperature sensors and vibration frequency sensors at the power line input end, heating elements and components that generate vibration of smart home appliances, the current, voltage, power, temperature and vibration frequency of smart home appliances are monitored in real time, and the multimodal data collected by these sensors can be used to evaluate and analyze the operating status of smart home appliances. By dynamically adjusting the acquisition frequency, the efficiency of data acquisition is improved: when the data change rate is small, the acquisition frequency is reduced, unnecessary data acquisition is reduced, and system resources and energy consumption are saved; when the data change rate is large, the acquisition frequency is increased to ensure the real-time and accuracy of the data; optimize system performance: dynamically adjust the acquisition frequency to reduce the burden on sensors and systems, extend the service life of equipment, and reduce maintenance costs; enhance the safety and reliability of smart home appliances, and by real-time monitoring of the multimodal data of smart home appliances, potential faults or abnormal conditions can be discovered and warned in time to prevent accidents and improve the safety and reliability of home appliances; improve user experience, and smart home appliances can be optimized and adjusted according to the collected data to improve operating efficiency and user satisfaction.

[0101] In one embodiment of the present invention, the multimodal data is cleaned and normalized, and after the cleaning and normalization, feature extraction is performed on the data of each modality, and the features extracted from each modality are combined into a comprehensive feature vector, including:

[0102] The multimodal data is cleaned and normalized, wherein the cleaning includes: processing missing values ​​and outliers, processing missing values ​​means filling missing values ​​with the previous valid values ​​collected by the sensor, processing outliers means determining whether the data has outliers with a preset first threshold, and if there are outliers, replacing them with the mean of the collected data, and the normalization process

[0103]

[0104] in, and are the mean and standard deviation of the data of sensor i, is the data collected by sensor i at time t, is the normalized data collected by sensor i at time t;

[0105] After cleaning and normalization, the following model is used to extract features from the data of each modality:

[0106]

[0107] in, represents the data of sensor i in the frequency domain, ω is the angular frequency, and represents the variables of different frequency components. is the normalized data collected by sensor i at time t, t represents the time variable, and j is a complex unit;

[0108]

[0109] in, represents the characteristics of the data of sensor i in the time-scale domain, a represents the scale parameter, b represents the displacement parameter, t represents the time variable, represents the mother wavelet function;

[0110] Combine the features extracted from each mode above into a comprehensive feature vector.

[0111]

[0112] in, represents the comprehensive feature vector of each sensor at time t, Represents the data of the current sensor in the frequency domain, Represents the data of the voltage sensor in the frequency domain, Represents the data of the power sensor in the frequency domain, Represents the characteristics of the current sensor data in the time-scale domain, Represents the characteristics of the voltage sensor data in the time-scale domain, Represents the characteristics of the power sensor data in the time-scale domain.

[0113] The working principle and effect of the above technical solution are: improving the reliability and accuracy of data, ensuring the integrity and consistency of data by filling missing values ​​and replacing outliers; after normalization, making data of different modes comparable, facilitating subsequent feature extraction and analysis; the diversity and comprehensiveness of feature extraction, combining the feature extraction methods of frequency domain and time-scale domain, can capture different feature information in the data and provide a more comprehensive feature representation; the construction of comprehensive feature vectors can effectively integrate the features of multimodal data, and improve the richness and information content of feature representation; suitable for multimodal data analysis and application: this solution is suitable for data processing and feature extraction of various sensors, providing high-quality feature input for subsequent machine learning models and data analysis; it can be widely used in industrial monitoring, intelligent manufacturing, Internet of Things and other fields to improve the accuracy and reliability of data-driven decision-making.

[0114] In one embodiment of the present invention, the comprehensive feature vector is input into a machine learning model for prediction, comprising:

[0115] The integrated feature vectors fused from the historical data are divided into a training set and a validation set. The training set is used to train the model by iteratively optimizing the parameters. The model performance is evaluated on the validation set, and the model parameters are adjusted to obtain a trained prediction model.

[0116] The newly fused comprehensive feature vector is input into the prediction model to generate a prediction result.

[0117]

[0118] in, is the predicted operating status of the home appliance. represents the comprehensive feature vector of each sensor at time t, represents the parameters of the prediction model, and f represents the prediction function;

[0119] When the prediction result is greater than a first preset threshold, an early warning message is sent to the user to remind the user of a potential fault.

[0120] The working principle and effect of the above technical solution are as follows: data partitioning, dividing the integrated feature vector after fusion in the historical data into a training set and a validation set; model training, using the training set to train the model by iteratively optimizing the parameters; evaluating the model performance on the validation set, and adjusting the model parameters according to the performance feedback; prediction and early warning, inputting the newly fused integrated feature vector into the trained prediction model to generate the prediction result; when the prediction result is greater than the first preset threshold, sending an early warning message to the user to remind the potential fault. Improve the prediction accuracy, through the division of the training set and the validation set and iterative optimization, the model can more accurately predict the operating status of the home appliance; timely early warning, when the prediction result shows a potential fault, the system can notify the user in time to take preventive measures; support decision-making, the prediction model can be used as an important tool in the smart home system to help users make better maintenance and use decisions; the application of this method is not limited to home appliances, but can also be extended to other fields, such as industrial equipment monitoring, intelligent manufacturing, etc., providing powerful data analysis and prediction capabilities.

[0121] In one embodiment of the present invention, after the early warning, historical maintenance records and service life are combined to generate maintenance suggestions, including:

[0122] when When the user is prompted to replace the content, a suggestion for immediate replacement is generated and sent to the user;

[0123] when When the repair is required, a suggestion is generated and sent to the user;

[0124] when When the system is running, it generates suggestions for normal operation and sends them to the user;

[0125] in, , represents the comprehensive score of historical maintenance records, n represents the number of maintenance times, is represented as the score of the ith maintenance, is the time of the ith maintenance, λ is the attenuation coefficient, indicating that the impact of historical maintenance gradually weakens over time; A= , T1 is the usage time of the smart appliance, the usage time of the smart appliance is the time interval between the current time and the installation time, T2 is the expected life span of the smart appliance, m1 is the preset second threshold, m2 is the preset third threshold, α, β and λ are weight coefficients.

[0126] The working principles and effects of the above technical solutions are as follows: Improved comprehensive evaluation accuracy: by combining prediction results, historical maintenance records and usage time ratios, a more comprehensive and accurate evaluation of the operating status of home appliances can be made; personalized maintenance suggestions: different maintenance suggestions (replacement, repair, normal) are generated according to different comprehensive scores, providing users with specific and actionable maintenance suggestions to improve user experience; timely early warning and maintenance: when potential faults are detected or repairs are required, users can be notified in time to reduce the inconvenience and losses caused by equipment failures; equipment service life is extended: timely maintenance and replacement can avoid excessive wear of equipment, extend the service life of equipment, and improve the economic benefits for users.

[0127] One embodiment of the present invention provides a smart home remote control and early warning system, the system comprising:

[0128] A data acquisition module is used to acquire multimodal data of the operation of smart appliances in a smart home;

[0129] A comprehensive feature vector generation module is used to clean and normalize the multimodal data, extract features from the data of each modality after cleaning and normalization, and combine the features extracted from each modality into a comprehensive feature vector;

[0130] An early warning module, used for inputting the comprehensive feature vector into a machine learning model for prediction, and issuing an early warning when the comprehensive feature vector exceeds a threshold;

[0131] The maintenance suggestion generation module is used to generate maintenance suggestions based on historical maintenance records and service life after early warning.

[0132] The working principle and effect of the above technical solution are as follows: multimodal data acquisition, smart home devices collect data through a variety of sensors, including but not limited to current, voltage, power, temperature, vibration frequency and other related environmental parameters, etc. These sensor data constitute multimodal data, which can fully reflect the operating status of home appliances; data cleaning and normalization,

[0133] The collected multimodal data is cleaned to remove noise and outliers to ensure the integrity and accuracy of the data; the normalization process standardizes the data so that the data of different modes have the same scale, which is convenient for subsequent analysis and processing; the cleaned and normalized multimodal data is subjected to feature extraction, and useful features are extracted from the data of each mode. The features extracted from each mode are combined into a comprehensive feature vector (including the features of all sensor data) for more comprehensive analysis; the comprehensive feature vector is input into the trained machine learning model for prediction. Commonly used models include long short-term memory network (LSTM), support vector machine (SVM), etc.; the model is used to predict the operating status of household appliances. When the prediction result of the comprehensive feature vector exceeds the set threshold, the system will issue a warning signal; the warning information is sent to the user through the smart home system to remind the user to pay attention to potential faults. According to the warning results, combined with the historical maintenance records and service life of the household appliances, maintenance recommendations are generated. It can monitor the operating status of smart home devices in real time, and comprehensively understand the health status of the equipment through comprehensive analysis of multimodal data; fault warning, through the prediction of machine learning models, it can give early warning before the fault occurs, reducing the loss and inconvenience caused by sudden equipment failure; data-driven, using multimodal data and machine learning technology, the warning system has high accuracy and robustness, can effectively identify potential faults, and improve the system's warning ability; maintenance suggestions, combined with historical maintenance records and years of use, generate accurate maintenance suggestions to help users maintain or replace equipment in time, extend the service life of equipment, and improve user experience; through automated monitoring and early warning, the smart home system reduces the need for manual inspection and maintenance, improves management efficiency, saves time and labor costs; through intelligent early warning and maintenance suggestions, users can use smart home devices more safely and improve the overall quality of life; this technical solution realizes real-time monitoring and fault warning of smart home devices through intelligent data monitoring and analysis, improves the security, reliability and user experience of the system, and has significant application value and promotion prospects.

[0134] In one embodiment of the present invention, the data acquisition module includes:

[0135] A current data acquisition module, wherein a current sensor disposed at the input end of the power line of the smart home appliance detects the current passing through the smart home appliance;

[0136] A voltage data acquisition module, wherein a voltage sensor disposed at the power input terminal of the smart home appliance detects the voltage passing through the smart home appliance;

[0137] A power data acquisition module, wherein a power sensor disposed at the power input terminal of the smart home appliance detects the power passing through the smart home appliance;

[0138] A temperature data acquisition module, wherein a temperature sensor disposed at a heating element of the smart home appliance detects the temperature of the smart home appliance;

[0139] A vibration frequency data acquisition module, wherein a vibration frequency sensor disposed at a component of the smart home appliance that generates vibration detects the vibration frequency of the smart home appliance;

[0140] The acquisition frequency calculation module collects the multimodal data according to the acquisition frequency. Specifically, the acquisition frequency model is:

[0141]

[0142] in, is the minimum acquisition frequency, is the maximum acquisition frequency, is the minimum value of the rate of change, is the maximum value of the rate of change, It is the rate of change of the data collected at time t relative to the previous time.

[0143] The working principle and effect of the above technical solution are as follows: by setting current sensors, voltage sensors, power sensors, temperature sensors and vibration frequency sensors at the power line input end, heating elements and components that generate vibration of smart home appliances, the current, voltage, power, temperature and vibration frequency of smart home appliances are monitored in real time, and the multimodal data collected by these sensors can be used to evaluate and analyze the operating status of smart home appliances. By dynamically adjusting the acquisition frequency, the efficiency of data acquisition is improved: when the data change rate is small, the acquisition frequency is reduced, unnecessary data acquisition is reduced, and system resources and energy consumption are saved; when the data change rate is large, the acquisition frequency is increased to ensure the real-time and accuracy of the data; optimize system performance: dynamically adjust the acquisition frequency to reduce the burden on sensors and systems, extend the service life of equipment, and reduce maintenance costs; enhance the safety and reliability of smart home appliances, and by real-time monitoring of the multimodal data of smart home appliances, potential faults or abnormal conditions can be discovered and warned in time to prevent accidents and improve the safety and reliability of home appliances; improve user experience, and smart home appliances can be optimized and adjusted according to the collected data to improve operating efficiency and user satisfaction.

[0144] In one embodiment of the present invention, the module for generating a comprehensive feature vector comprises:

[0145] The preprocessing module is used to clean and normalize the multimodal data. The cleaning includes: processing missing values ​​and outliers. Processing missing values ​​is to fill the missing values ​​with the previous valid values ​​collected by the sensor. Processing outliers is to determine whether the data has outliers with a preset first threshold. If there are outliers, replace them with the mean of the collected data. The normalization processing model is:

[0146]

[0147] in, and are the mean and standard deviation of the data of sensor i, is the data collected by sensor i at time t, is the normalized data collected by sensor i at time t;

[0148] The feature extraction module is used to extract features from the data of each modality using the following model after cleaning and normalization.

[0149]

[0150] in, represents the data of sensor i in the frequency domain, ω is the angular frequency, and represents the variables of different frequency components. is the normalized data collected by sensor i at time t, t represents the time variable, and j is a complex unit;

[0151]

[0152] in, represents the characteristics of the data of sensor i in the time-scale domain, a represents the scale parameter, b represents the displacement parameter, t represents the time variable, represents the mother wavelet function;

[0153] The combined feature vector module is used to combine the features extracted from each mode into a comprehensive feature vector.

[0154]

[0155] in, represents the comprehensive feature vector of each sensor at time t, Represents the data of the current sensor in the frequency domain, Represents the data of the voltage sensor in the frequency domain, Represents the data of the power sensor in the frequency domain, Represents the characteristics of the current sensor data in the time-scale domain, Represents the characteristics of the voltage sensor data in the time-scale domain, Represents the characteristics of the power sensor data in the time-scale domain.

[0156] The working principle and effect of the above technical solution are: improving the reliability and accuracy of data, ensuring the integrity and consistency of data by filling missing values ​​and replacing outliers; after normalization, making data of different modes comparable, facilitating subsequent feature extraction and analysis; the diversity and comprehensiveness of feature extraction, combining the feature extraction methods of frequency domain and time-scale domain, can capture different feature information in the data and provide a more comprehensive feature representation; the construction of comprehensive feature vectors can effectively integrate the features of multimodal data, and improve the richness and information content of feature representation; suitable for multimodal data analysis and application: this solution is suitable for data processing and feature extraction of various sensors, providing high-quality feature input for subsequent machine learning models and data analysis; it can be widely used in industrial monitoring, intelligent manufacturing, Internet of Things and other fields to improve the accuracy and reliability of data-driven decision-making.

[0157] In one embodiment of the present invention, the early warning module includes:

[0158] The training model module is used to divide the integrated feature vectors fused in the historical data into a training set and a validation set, use the training set to train the model by iteratively optimizing the parameters, evaluate the model performance on the validation set, and adjust the model parameters to obtain a trained prediction model;

[0159] A prediction result generating module is used to input the newly fused comprehensive feature vector into the prediction model to generate a prediction result.

[0160]

[0161] in, is the predicted operating status of the home appliance. represents the comprehensive feature vector of each sensor at time t, represents the parameters of the prediction model, and f represents the prediction function;

[0162] The module for sending early warning information is used to send early warning information to the user to remind the user of potential failure when the prediction result is greater than a first preset threshold.

[0163] The working principle and effect of the above technical solution are as follows: data partitioning, dividing the integrated feature vector after fusion in the historical data into a training set and a validation set; model training, using the training set to train the model by iteratively optimizing the parameters; evaluating the model performance on the validation set, and adjusting the model parameters according to the performance feedback; prediction and early warning, inputting the newly fused integrated feature vector into the trained prediction model to generate the prediction result; when the prediction result is greater than the first preset threshold, sending an early warning message to the user to remind the potential fault. Improve the prediction accuracy, through the division of the training set and the validation set and iterative optimization, the model can more accurately predict the operating status of the home appliance; timely early warning, when the prediction result shows a potential fault, the system can notify the user in time to take preventive measures; support decision-making, the prediction model can be used as an important tool in the smart home system to help users make better maintenance and use decisions; the application of this method is not limited to home appliances, but can also be extended to other fields, such as industrial equipment monitoring, intelligent manufacturing, etc., providing powerful data analysis and prediction capabilities.

[0164] In one embodiment of the present invention, the maintenance suggestion generating module includes:

[0165] Replace the module immediately for When the user is prompted to replace the content, a suggestion for immediate replacement is generated and sent to the user;

[0166] Repair module for When the repair is required, a suggestion is generated and sent to the user;

[0167] Normal operation module, used when When the system is running, it generates suggestions for normal operation and sends them to the user;

[0168] in, , represents the comprehensive score of historical maintenance records, n represents the number of maintenance times, is represented as the score of the ith maintenance, is the time of the ith maintenance, λ is the attenuation coefficient, indicating that the impact of historical maintenance gradually weakens over time; A= , T1 is the usage time of the smart appliance, the usage time of the smart appliance is the time interval between the current time and the installation time, T2 is the expected life span of the smart appliance, m1 is the preset second threshold, m2 is the preset third threshold, α, β and λ are weight coefficients.

[0169] The working principles and effects of the above technical solutions are as follows: Improved comprehensive evaluation accuracy: by combining prediction results, historical maintenance records and usage time ratios, a more comprehensive and accurate evaluation of the operating status of home appliances can be made; personalized maintenance suggestions: different maintenance suggestions (replacement, repair, normal) are generated according to different comprehensive scores, providing users with specific and actionable maintenance suggestions to improve user experience; timely early warning and maintenance: when potential faults are detected or repairs are required, users can be notified in time to reduce the inconvenience and losses caused by equipment failures; equipment service life is extended: timely maintenance and replacement can avoid excessive wear of equipment, extend the service life of equipment, and improve the economic benefits for users.

[0170] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A smart home remote control early warning method, characterized in that: The method comprises: Obtain multimodal data on the operation of smart appliances in smart homes; The multimodal data is cleaned and normalized, and after the cleaning and normalization, features of each modality data are extracted, and the features extracted from each modality are combined into a comprehensive feature vector, including: The multimodal data is cleaned and normalized. The cleaning includes: processing missing values ​​and outliers. Processing missing values ​​is to fill the missing values ​​with the previous valid values ​​collected by the sensor. Processing outliers is to determine whether the data has outliers with a preset first threshold. If there are outliers, replace them with the mean of the collected data. The normalization process is: in, and are the mean and standard deviation of the data of sensor i, is the data collected by sensor i at time t, is the normalized data collected by sensor i at time t; After cleaning and normalization, the following model is used to extract features from the data of each modality: in, represents the data of sensor i in the frequency domain, ω is the angular frequency, and represents the variables of different frequency components. is the normalized data collected by sensor i at time t, t represents the time variable, and j is a complex unit; in, represents the characteristics of the data of sensor i in the time-scale domain, a represents the scale parameter, b represents the displacement parameter, t represents the time variable, represents the mother wavelet function; Combine the features extracted from each mode above into a comprehensive feature vector. in, represents the comprehensive feature vector of each sensor at time t, Represents the data of the current sensor in the frequency domain, Represents the data of the voltage sensor in the frequency domain, Represents the data of the power sensor in the frequency domain, Represents the characteristics of the current sensor data in the time-scale domain, Represents the characteristics of the voltage sensor data in the time-scale domain, Representing the characteristics of the power sensor data in the time-scale domain; The comprehensive feature vector is input into a machine learning model for prediction to obtain a predicted home appliance operation state, and when the predicted home appliance operation state exceeds a threshold, an early warning is issued, including: The integrated feature vectors fused from the historical data are divided into a training set and a validation set. The training set is used to train the model by iteratively optimizing the parameters. The model performance is evaluated on the validation set, and the model parameters are adjusted to obtain a trained prediction model. The newly fused comprehensive feature vector is input into the prediction model to generate the prediction result ; When the prediction result is greater than a first preset threshold, a warning message is sent to the user to remind the user of a potential fault; After the warning, maintenance suggestions are generated based on historical maintenance records and service life, including: when When the user is prompted to replace the content, a suggestion for immediate replacement is generated and sent to the user; when When the repair is required, a suggestion is generated and sent to the user; when When the system is running, it generates suggestions for normal operation and sends them to the user; in, , represents the comprehensive score of historical maintenance records, n represents the number of maintenance times, is represented as the score of the ith maintenance, is the time of the ith maintenance, λ is the attenuation coefficient, indicating that the impact of historical maintenance gradually weakens over time; A= , T1 is the usage time of the smart appliance, the usage time of the smart appliance is the time interval between the current time and the installation time, T2 is the expected life span of the smart appliance, m1 is the preset second threshold, m2 is the preset third threshold, α, β and λ are weight coefficients.

2. According to claim 1, a smart home remote control early warning method is characterized in that: Obtain multimodal data on the operation of smart appliances in smart homes, including: A current sensor disposed at an input end of a power line of the smart home appliance detects a current passing through the smart home appliance; A voltage sensor disposed at a power input terminal of the smart home appliance detects a voltage passing through the smart home appliance; A power sensor disposed at the power input terminal of the smart home appliance detects the power passing through the smart home appliance; A temperature sensor disposed at a heating element of the smart home appliance detects the temperature of the smart home appliance; A vibration frequency sensor disposed at a component of the smart home appliance that generates vibration detects the vibration frequency of the smart home appliance; The multimodal data is collected according to the collection frequency. Specifically, the collection frequency model is: in, is the minimum acquisition frequency, is the maximum acquisition frequency, is the minimum value of the rate of change, is the maximum value of the rate of change, It is the rate of change of the data collected at time t relative to the previous time.

3. A smart home remote control and early warning system, characterized in that: The system comprises: A data acquisition module is used to acquire multimodal data of the operation of smart appliances in a smart home; A comprehensive feature vector generation module is used to clean and normalize the multimodal data, extract features from the data of each modality after cleaning and normalization, and combine the features extracted from each modality into a comprehensive feature vector; An early warning module, used for inputting the comprehensive feature vector into a machine learning model for prediction, and issuing an early warning when the comprehensive feature vector exceeds a threshold; The maintenance suggestion generation module is used to generate maintenance suggestions based on historical maintenance records and service life after early warning; The module for generating a comprehensive feature vector comprises: The preprocessing module is used to clean and normalize the multimodal data. The cleaning includes: processing missing values ​​and outliers. Processing missing values ​​is to fill the missing values ​​with the previous valid values ​​collected by the sensor. Processing outliers is to determine whether the data has outliers with a preset first threshold. If there are outliers, replace them with the mean of the collected data. The normalization process is: in, and are the mean and standard deviation of the data of sensor i, is the data collected by sensor i at time t, is the normalized data collected by sensor i at time t; The feature extraction module is used to extract features from the data of each modality using the following model after cleaning and normalization. in, represents the data of sensor i in the frequency domain, ω is the angular frequency, and represents the variables of different frequency components. is the normalized data collected by sensor i at time t, t represents the time variable, and j is a complex unit; in, represents the characteristics of the data of sensor i in the time-scale domain, a represents the scale parameter, b represents the displacement parameter, t represents the time variable, represents the mother wavelet function; The combined feature vector module is used to combine the features extracted from each mode into a comprehensive feature vector. in, represents the comprehensive feature vector of each sensor at time t, Represents the data of the current sensor in the frequency domain, Represents the data of the voltage sensor in the frequency domain, Represents the data of the power sensor in the frequency domain, Represents the characteristics of the current sensor data in the time-scale domain, Represents the characteristics of the voltage sensor data in the time-scale domain, Representing the characteristics of the power sensor data in the time-scale domain; The early warning module comprises: The training model module is used to divide the integrated feature vectors fused in the historical data into a training set and a validation set, use the training set to train the model by iteratively optimizing the parameters, evaluate the model performance on the validation set, and adjust the model parameters to obtain a trained prediction model; Generate prediction result module, used to input the newly integrated feature vector into the prediction model to generate prediction results ; A warning information sending module is used to send a warning message to a user to remind a potential fault when the prediction result is greater than a first preset threshold; The maintenance suggestion generating module comprises: Replace the module immediately for When the user is prompted to replace the content, a suggestion for immediate replacement is generated and sent to the user; Repair module for When the repair is required, a suggestion is generated and sent to the user; Normal operation module, used when When the system is running, it generates suggestions for normal operation and sends them to the user; in, , represents the comprehensive score of historical maintenance records, n represents the number of maintenance times, is represented as the score of the ith maintenance, is the time of the ith maintenance, λ is the attenuation coefficient, indicating that the impact of historical maintenance gradually weakens over time; A= , T1 is the usage time of the smart appliance, the usage time of the smart appliance is the time interval between the current time and the installation time, T2 is the expected life span of the smart appliance, m1 is the preset second threshold, m2 is the preset third threshold, α, β and λ are weight coefficients.

4. According to claim 3, a smart home remote control and early warning system is characterized in that: The data acquisition module includes: A current data acquisition module, wherein a current sensor disposed at the input end of the power line of the smart home appliance detects the current passing through the smart home appliance; A voltage data acquisition module, wherein a voltage sensor disposed at the power input terminal of the smart home appliance detects the voltage passing through the smart home appliance; A power data acquisition module, wherein a power sensor disposed at the power input terminal of the smart home appliance detects the power passing through the smart home appliance; A temperature data acquisition module, wherein a temperature sensor disposed at a heating element of the smart home appliance detects the temperature of the smart home appliance; A vibration frequency data acquisition module, wherein a vibration frequency sensor disposed at a component of the smart home appliance that generates vibration detects the vibration frequency of the smart home appliance; The acquisition frequency calculation module collects the multimodal data according to the acquisition frequency. Specifically, the acquisition frequency model is: in, is the minimum acquisition frequency, is the maximum acquisition frequency, is the minimum value of the rate of change, is the maximum value of the rate of change, It is the rate of change of the data collected at time t relative to the previous time.

Images