A method and system for detecting abnormal operation state of a freezer based on speech recognition and internet of things

Abstract

The application discloses an ice cabinet running state abnormality detection method and system based on voice recognition and the Internet of Things, comprising collecting sound samples of the ice cabinet under normal working state and abnormal state, and pre-processing and feature extraction of the sound samples; constructing a structured ice cabinet sound feature library, and storing the extracted features in association with corresponding running state labels and associated IoT data; constructing a multi-dimensional factor regression model to analyze the influence of multi-dimensional factors on the result; constructing a multi-modal ice cabinet abnormality detection model; training the multi-modal ice cabinet abnormality detection model through the sound samples; inputting the collected data into the abnormality detection model, and determining whether the current running state is abnormal by the model; if the current running state is abnormal, outputting an abnormality type and a confidence level; when the abnormality is detected and the confidence level exceeds a set threshold, triggering a warning, and pushing the warning information to a user terminal APP and / or a cloud service platform. The application has a high fault recognition rate, realizes early fault warning, and avoids deterioration of the goods in the cabinet.

Description

Technical Field

[0001] This invention relates to the field of smart home appliance technology, and in particular to a method and system for detecting abnormal operating status of a freezer based on voice recognition and the Internet of Things. Background Technology

[0002] Freezers, as core cold chain equipment, are widely used in many fields such as food processing and storage, medical drug preservation, biological sample storage, supermarket retail, catering services, and households. Their stable and reliable operation is crucial for ensuring the quality, safety, and value of goods (especially perishable foods, vaccines, reagents, etc.).

[0003] However, during long-term operation, freezers are highly susceptible to malfunctions due to factors such as equipment aging, component failures (e.g., compressor malfunction, refrigerant leakage, fan failure, door seal aging), improper use (e.g., frequent door opening, items blocking airflow), unstable power supply, or environmental factors (high temperature, high humidity). If these malfunctions are not detected and addressed promptly, they can lead to serious consequences. At best, the internal temperature may exceed safe limits, causing spoilage and economic losses. At worst, they could result in food safety incidents, medical accidents, damage to research samples, or even fires due to continuous abnormal operation (e.g., compressor overheating), with extremely severe consequences.

[0004] Currently, the monitoring of freezer operation status mainly relies on the following methods: (1) Manual periodic inspection and recording: Staff regularly check the real-time temperature displayed by the freezer and manually record it. This method has high labor costs, low efficiency and poor real-time performance. Abnormalities occurring during the inspection interval cannot be detected in time, and manual reading and recording are prone to errors and omissions, especially for widely distributed freezer groups, which are extremely difficult to manage. (2) Basic sensor threshold alarm: Temperature sensors are installed in the freezer, and when the detected temperature exceeds the preset upper or lower threshold, an alarm is triggered (such as an audible and visual alarm or a simple remote notification). This is currently the most widely used method. However, its main disadvantage is that the response is delayed, and the alarm is usually triggered when the temperature has deviated significantly from the set value (the items may have already begun to be damaged), which cannot achieve early warning. The false alarm rate is high. Brief door opening, instantaneous power outage recovery, sensor malfunction or environmental interference can easily lead to unnecessary false alarms, interfering with normal operation and maintenance. There is a risk of underreporting. This method cannot effectively identify anomalies not directly related to temperature (such as a sharp increase in energy consumption due to decreased compressor efficiency but the temperature has not yet exceeded the limit, the fan stopping but the refrigerant is still in the early stage of circulation and the temperature has not risen significantly, or a slow temperature rise caused by a door not being closed tightly). Furthermore, the information is limited, relying solely on the temperature parameter, making it difficult to accurately determine the root cause of the fault. Summary of the Invention

[0005] In order to overcome the above-mentioned problems in the existing technology, the present invention proposes a method and system for detecting abnormal operation status of freezers based on voice recognition and the Internet of Things.

[0006] The technical solution adopted by this invention to solve its technical problem is: a method for detecting abnormal operating status of a freezer based on voice recognition and the Internet of Things, comprising the following steps:

[0007] Step 1: Collect sound samples from the freezer under normal operating conditions and under abnormal conditions, and preprocess and extract features from them;

[0008] Step 2: Based on the data obtained in Step 1, construct a structured freezer sound feature library, and associate and store the extracted features with the corresponding operating status labels and associated IoT data;

[0009] Step 3: Construct a multidimensional factor regression model and analyze the impact of multidimensional factors on the results;

[0010] Step 4: Construct a multimodal freezer anomaly detection model. Based on the analysis results of Step 3, introduce high-weight factors into the multimodal freezer detection model.

[0011] Step 5: Train the multimodal freezer anomaly detection model from Step 4 using the sound samples obtained in Step 2;

[0012] Step 6: Collect the sound of the freezer running in real time through the built-in microphone array and collect the freezer's operating status information through the built-in sensors. Input the above information into the model obtained in Step 5. The model determines whether the current operating status is abnormal based on the input data. If abnormal, it outputs the abnormality type and confidence level.

[0013] Step 7: When an anomaly is detected and the confidence level exceeds the set threshold, an alert is triggered and the alert information is pushed to the user terminal APP and / or cloud service platform.

[0014] The above-mentioned method for detecting abnormal operation status of freezers based on speech recognition and the Internet of Things includes, in step 1, feature extraction, which specifically includes: spectral features, time-domain features, time-frequency domain features, deep learning-based embedded features, and abstract features extracted using RNN.

[0015] The above-mentioned method for detecting abnormal operating status of freezers based on voice recognition and the Internet of Things, specifically step 3 is as follows:

[0016] Step 3.1: Select a multiple linear regression model and determine its dimensions;

[0017] Step 3.2: Calculate the predicted values ​​of the model, using the mean squared error as the loss function;

[0018] Step 3.3: Minimize the loss function using the gradient descent algorithm;

[0019] Step 3.4: Repeat steps 3.2-3.3 until the value of the loss function drops below the preset threshold, or the rate of decrease in the loss function becomes smaller, indicating that the model has converged.

[0020] Step 3.5: Predict the target value on the test dataset, and judge the predictive ability of the model based on the average error between the predicted actual value and the predicted target value.

[0021] The above-mentioned method for detecting abnormal operating status of freezers based on speech recognition and the Internet of Things, specifically step 3.5 of predicting the target value on the test dataset involves defining a multi-dimensional scoring model, the specific expression of which is:

[0022]

[0023]

[0024] in, For user u, predict the multi-dimensional results for project i, where N is the set of attributes. This represents the attribute value of attribute factor f. The weight of attribute f. For user u, the prediction under attribute f for item i.

[0025] A freezer operation anomaly detection system based on voice recognition and the Internet of Things (IoT) employs the aforementioned method for detecting freezer operation anomalies based on voice recognition and IoT. The system includes a freezer body, a sensor array, a microphone array, a data processing module, a communication module, a cloud server, and an algorithm storage module. The sensor array and microphone array are used to collect freezer operation data. The data processing module preprocesses the sound data collected by the microphone array. The preprocessed sound data and the data collected by the sensor array are input into the algorithm storage module, which runs an anomaly detection model. The freezer body is connected to the cloud server via the communication module.

[0026] The above-mentioned abnormal operation status detection system for freezers based on voice recognition and the Internet of Things has an algorithm storage module located on a cloud server. The pre-processed sound data and the data collected by the sensor group are uploaded to the cloud server through the communication module, and the algorithm storage module runs an abnormal detection model to perform detection.

[0027] The above-mentioned abnormal operation status detection system for a freezer based on voice recognition and the Internet of Things has an algorithm storage module built into the freezer body. The pre-processed sound data and the data collected by the sensor group are input into the algorithm storage module to realize the fusion analysis of sound features and IoT data locally. When the confidence level output by the abnormal detection model is lower than the set threshold, the sound and light alarm of the freezer body is triggered. At the same time, the abnormal event result information is uploaded to the cloud server through the communication module.

[0028] The beneficial effects of this invention are: early warning: by capturing and analyzing predictive abnormal sounds, potential faults can be detected before mechanical parts completely fail or the temperature rises significantly, greatly advancing the warning time and avoiding food loss.

[0029] Precise diagnosis: By combining sound characteristics and IoT data, it can not only determine that "there is an anomaly", but also make a preliminary diagnosis of the anomaly type (such as distinguishing between compressor failure, fan failure, door seal problem, refrigerant problem, etc.), providing key information for users or maintenance personnel.

[0030] Reduce false alarms: Integrating IoT data (such as door open / close status and ambient temperature) helps distinguish between genuine faults and environmental interference (such as a temporary temperature rise caused by opening the door, or a compressor running continuously at high ambient temperature and making a slightly louder noise).

[0031] Non-invasive monitoring: It uses a microphone to collect sound, eliminating the need for additional dedicated sensors on critical mechanical components, reducing costs, and making it easy to deploy on existing or new freezers.

[0032] High level of intelligence: It uses AI models for automatic analysis, reducing the burden on users.

[0033] Enhance user experience and sense of security: Users can remotely check the health status of the freezer, address issues promptly, and reduce anxiety and losses.

[0034] Optimized maintenance: Provides pre-diagnostic information for after-sales service, improving repair efficiency. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the process of this invention;

[0036] Figure 2 This is a schematic diagram of the real-time monitoring and early warning process for abnormal detection of the freezer in this invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1As shown in the figure, this embodiment discloses a method for detecting abnormal operating status of a freezer based on voice recognition and the Internet of Things, including the following steps:

[0039] Step 1: Collect sound samples of the freezer under normal operating conditions (e.g., compressor start-up, stable operation, defrosting, operation under different ambient temperatures) and abnormal conditions (e.g., abnormal noise / overload / start-up failure of the compressor, fan jamming / abnormal noise, abnormal airflow noise caused by refrigerant leakage / insufficiency, continuous operating noise caused by poor door seal) and preprocess them (noise reduction, filtering) and extract features.

[0040] The key features extracted include:

[0041] Spectral characteristics: energy distribution, spectral centroid, spectral bandwidth, harmonic characteristics, etc. in a specific frequency band.

[0042] Temporal characteristics: volume (sound pressure level), zero-crossing rate, energy envelope, etc.

[0043] Time-frequency domain characteristics: MFCC (Mel frequency cepstral coefficients), chromaticity characteristics, etc.

[0044] Advanced features: Deep learning-based embedding features are abstract features extracted using RNNs.

[0045] Step 2: Based on the data obtained in Step 1, construct a structured freezer sound feature library, and associate and store the extracted features with the corresponding operating status labels (normal / abnormal types) and associated IoT data (such as real-time temperature, compressor start / stop status, door open / close status, and ambient temperature).

[0046] Step 3: Construct a multidimensional factor regression model and analyze the impact of multidimensional factors on the results.

[0047] Step 3 specifically involves:

[0048] Step 3.1: Select a multiple linear regression model and determine its dimensions. The multiple linear regression model takes the following form:

[0049]

[0050] Where ε is the random error. , … For regression coefficients, y is the independent variable; y is the dependent variable.

[0051] Step 3.2: Calculate the model's predicted values, using mean squared error (MSE) as the loss function.

[0052] = .

[0053] The goal of model training is to find a set of optimal regression coefficients that minimize the model's prediction error and achieve the best fit on the training data. Assuming there are m samples, each with n features, the optimal model parameters (loss function) are used to measure the error between the predicted and actual values, with mean squared error (MSE) used as the loss function.

[0054]

[0055] in, , … For regression coefficients, It is the predicted value for all samples.

[0056] Step 3.3: Minimize the loss function using the gradient descent algorithm.

[0057] Backpropagation uses gradient descent to minimize the loss function. First, it calculates the partial derivative of each regression coefficient in the above equation:

[0058] =

[0059] in This is the j-th feature value of the i-th sample. Then, gradient descent is used to update the regression coefficients.

[0060] = -

[0061] in It is the learning rate, used to control the update step size.

[0062] Step 3.4: Repeat steps 3.2-3.3 until the value of the loss function drops below the preset threshold, or the rate of decrease in the loss function becomes smaller, indicating that the model has converged.

[0063] Step 3.5: Predict the target value on the test dataset, and judge the predictive ability of the model based on the average error between the predicted actual value and the predicted target value.

[0064] Step 3.5, predicting the target value on the test dataset, specifically involves: assuming n valid attribute factors are obtained, recording the correlation coefficients of each influencing factor attribute. , …, Define the weights of each attribute factor as follows: , ,…, Therefore, the scoring model for each attribute dimension is defined as follows:

[0065]

[0066] Where N is the set of attributes, This represents the attribute value of attribute factor f. The weight of attribute f. For user u, the prediction under attribute f for item i.

[0067] The multi-dimensional scoring model is defined as follows:

[0068]

[0069] It can be simplified to:

[0070]

[0071] in For user u, predict the multi-dimensional results for project i.

[0072] Model evaluation uses a test dataset to assess the regression model's capability by measuring the average error between the predicted and actual values ​​based on the mean squared error (MSE).

[0073] In this embodiment, experiments revealed that factors such as door opening / closing status, temperature, voltage, current, and power are all high-weight influencing factors.

[0074] Step 4: Construct a multimodal freezer detection model. Based on the analysis results of Step 3, introduce high-weight factors into the multimodal freezer detection model.

[0075] A classification / anomaly detection model based on machine learning and / or deep learning. The core inputs to this model include:

[0076] Real-time sound features: Features extracted from the real-time audio stream captured by the built-in microphone of the freezer.

[0077] Real-time IoT data: Real-time data provided by freezer sensors (such as internal temperature, set temperature, compressor operating status, door open / close status, ambient temperature, etc.).

[0078] The model's output includes:

[0079] Abnormal status judgment: Normal / Abnormal.

[0080] Anomaly type prediction: (optional) Predict possible anomaly types (such as "compressor noise", "fan failure", "insufficient cooling", "door seal abnormality", etc.).

[0081] Confidence level: The level of confidence in the judgment result.

[0082] Step 5: Train the multimodal freezer detection model from Step 4 using the sound samples obtained in Step 2.

[0083] Step 6: The freezer's built-in microphone array collects real-time audio during operation, and the freezer's built-in sensors collect its operating status information. This information is input into the model obtained in Step 5. The model determines whether the current operating status is abnormal based on the input data. If abnormal, it outputs the abnormality type and confidence level. Figure 2 As shown.

[0084] Step 7: When an anomaly is detected and the confidence level exceeds the set threshold, an alert is triggered and the alert information is pushed to the user terminal APP and / or cloud service platform.

[0085] When an anomaly is detected and the confidence level exceeds a set threshold, the system triggers an alert: the alert information (including the anomaly status, type, confidence level, associated sensor data snapshots, and possible timestamped audio clips) is pushed to the user terminal APP and / or cloud service platform via the IoT network.

[0086] The user app or cloud platform can provide more detailed diagnostic suggestions (such as "Please check if the compressor is making abnormal noises", "The door may not be closed properly", "It is recommended to contact after-sales service to check the refrigeration system", and for serious cases, directly push the after-sales platform to dispatch a technician to the site). The user app can also provide user interaction: The user app displays warning information, provides operation options such as "confirm", "ignore", and "contact customer service", and can view historical abnormal records.

[0087] Example 1

[0088] Based on the above-mentioned anomaly detection method, this embodiment also discloses a freezer operation status anomaly detection system based on voice recognition and the Internet of Things, including a freezer body, a sensor group (temperature sensor, door magnetic sensor, compressor status sensor, fan status sensor, etc.) installed on the freezer body, a microphone array, a data processing module, a communication module, a cloud server, and an algorithm storage module. The sensor group and microphone array are used to collect freezer body operation data. The data processing module is used to preprocess the sound data collected by the microphone array. The preprocessed sound data and the data collected by the sensor group are input into the algorithm storage module. The algorithm storage module runs an anomaly detection model. The freezer body is connected to the cloud server through the communication module.

[0089] The algorithm storage module is located on a cloud server. The pre-processed sound data and the data collected by the sensor group are uploaded to the cloud server through the communication module. The algorithm storage module runs an anomaly detection model to perform detection.

[0090] In this embodiment, the anomaly detection model is a pruned and quantized MobileNet or a small Transformer model.

[0091] Example 2

[0092] Based on the above-mentioned anomaly detection method, this embodiment also discloses a freezer operation status anomaly detection system based on voice recognition and the Internet of Things. The difference from Embodiment 1 is that the algorithm storage module is built into the freezer body. The pre-processed sound data and the data collected by the sensor group are input into the algorithm storage module to realize the fusion analysis of sound features and IoT data locally. When the confidence level output by the anomaly detection model is lower than the set threshold, the sound and light alarm of the freezer body is triggered. At the same time, the abnormal event result information is uploaded to the cloud server through the communication module.

[0093] In this embodiment, the anomaly detection model is a pruned and quantized MobileNet or a small Transformer model.

[0094] This method only reports the results of abnormal events (type, confidence level, key data snapshots) to the cloud and user apps, significantly reducing network transmission volume and cloud load, improving response speed, and protecting privacy.

[0095] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its scope and spirit, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A method for detecting abnormal operating status of a freezer based on speech recognition and the Internet of Things, characterized in that, Includes the following steps: Step 1: Collect sound samples from the freezer under normal operating conditions and under abnormal conditions, and preprocess and extract features from them; Step 2: Based on the data obtained in Step 1, construct a structured freezer sound feature library, and associate and store the extracted features with the corresponding operating status labels and associated IoT data; Step 3: Construct a multidimensional factor regression model and analyze the impact of multidimensional factors on the results; Step 4: Construct a multimodal freezer anomaly detection model. Based on the analysis results of Step 3, introduce high-weight factors into the multimodal freezer anomaly detection model. Step 5: Train the multimodal freezer anomaly detection model from Step 4 using the sound samples obtained in Step 2; Step 6: Collect the sound of the freezer running in real time through the built-in microphone array and collect the freezer's operating status information through the built-in sensors. Input the above information into the model obtained in Step 5. The model determines whether the current operating status is abnormal based on the input data. If abnormal, it outputs the abnormality type and confidence level. Step 7: When an anomaly is detected and the confidence level exceeds the set threshold, an alert is triggered and the alert information is pushed to the user terminal APP and / or cloud service platform. Step 3 specifically involves: Step 3.1: Select a multiple linear regression model and determine its dimensions; Step 3.2: Calculate the predicted values ​​of the model, using the mean squared error as the loss function; Step 3.3: Minimize the loss function using the gradient descent algorithm; Step 3.4: Repeat steps 3.2-3.3 until the value of the loss function drops below the preset threshold, or the rate of decrease in the loss function becomes smaller, indicating that the model has converged. Step 3.5: Predict the target value on the test dataset, and judge the predictive ability of the model based on the average error between the predicted actual value and the predicted target value. In step 3.5, predicting the target value on the test dataset specifically involves defining a multi-dimensional scoring model, the specific expression of which is: in, For user u, predict the multi-dimensional results for project i, where N is the set of attributes. This represents the attribute value of attribute factor f. The weight of attribute f. For user u, the prediction under attribute f for item i.

2. The method for detecting abnormal operating status of a freezer based on voice recognition and the Internet of Things according to claim 1, characterized in that, The feature extraction in step 1 specifically includes: spectral features, time-domain features, time-frequency domain features, deep learning-based embedding features, and abstract features extracted using RNN.

3. A freezer operation status anomaly detection system based on voice recognition and the Internet of Things, characterized in that, The method for detecting abnormal operating status of a freezer based on voice recognition and the Internet of Things, as described in any one of claims 1-2, includes a freezer body, a sensor group installed on the freezer body, a microphone array, a data processing module, a communication module, a cloud server, and an algorithm storage module. The sensor group and microphone array are used to collect operating data of the freezer body. The data processing module is used to preprocess the sound data collected by the microphone array. The preprocessed sound data and the data collected by the sensor group are input into the algorithm storage module. The algorithm storage module runs an abnormal detection model. The freezer body is connected to the cloud server through the communication module.

4. The freezer operation status anomaly detection system based on voice recognition and the Internet of Things as described in claim 3, characterized in that, The algorithm storage module is located on a cloud server. The pre-processed sound data and the data collected by the sensor group are uploaded to the cloud server through the communication module. The algorithm storage module runs an anomaly detection model to perform detection.

5. A freezer operation status anomaly detection system based on voice recognition and the Internet of Things as described in claim 3, characterized in that, The algorithm storage module is built into the freezer body. The pre-processed sound data and the data collected by the sensor group are input into the algorithm storage module to realize the fusion analysis of sound features and IoT data locally. When the confidence level output by the anomaly detection model is lower than the set threshold, the sound and light alarm of the freezer body is triggered. At the same time, the abnormal event result information is uploaded to the cloud server through the communication module.

Images