Smart home early warning system based on multi-source data fusion

Abstract

The invention discloses a smart home early warning system based on multi-source data fusion, relates to the technical field of AI large models, and solves the problems that the temperature change in the internal environment of a refrigerator is possible to be fast, and the complex condition in the refrigerator is difficult to detect only through food images, so that the early warning result of the system is not accurate enough, and the early warning effect is poor. Therefore, the food safety performance is reduced. The state recognition module is used for performing analysis based on a plurality of refrigerator loads to obtain refrigerator states; the data acquisition module is used for acquiring external environment temperature and internal image data of the target refrigerator; the data analysis module is used for analyzing the refrigerator load based on the environment temperature to obtain theoretical load capacity; various food types in the refrigerator are obtained based on image data analysis, and the food types are obtained and analyzed to obtain theoretical food heat energy; the intelligent early warning module is used for obtaining the refrigerator load capacity and generating an early warning signal based on the refrigerator load capacity, the theoretical load capacity and the theoretical heat energy; and the early warning result accuracy is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of multi-source data fusion, and specifically to a smart home early warning system based on multi-source data fusion. Background Art

[0002] AI big model refers to the process of integrating data from different sources, modalities, formats, or time and space, and collaboratively processing and analyzing them using algorithms and technical means to extract more comprehensive, accurate, and reliable information or knowledge;

[0003] Prior art (CN119492234A) provides an artificial intelligence-based food storage early warning method and system. The method includes: obtaining an internal image of a target refrigerator, multi-source sensor signals, and the time each food item was placed in the refrigerator; performing image denoising and enhancement processing on the internal image to obtain a preprocessed image; performing image recognition based on the preprocessed image to obtain food information for each type of food item; the food information includes: food type, number of items, and food color; performing signal analysis on the multi-source sensor signals to obtain temperature information, humidity information, and gas information; inputting the placement time, temperature information, humidity information, gas information, and food information into a preset food freshness prediction model to obtain a prediction result; and issuing an early warning based on the prediction result. The present invention can intelligently monitor food freshness and issue timely early warnings, thereby effectively reducing food waste and improving the efficiency and safety of food management.

[0004] The above-mentioned artificial intelligence-based food storage warning method and system inputs time, temperature information, humidity information, gas information and food information into a preset food freshness prediction model to obtain a prediction result, and issues a warning based on the prediction result. It can intelligently monitor the freshness of food and issue a warning in time; however, in actual applications, if there is an abnormality in the refrigerator's refrigeration system or insufficient power supply, the temperature inside the refrigerator may change rapidly. It is difficult to detect the above-mentioned complex conditions inside the refrigerator only through food images, and false alarms may occur, resulting in inaccurate system warning results, thereby reducing food safety performance. Summary of the Invention

[0005] The present application aims to solve at least one of the technical problems existing in the prior art; to this end, the present application proposes an intelligent home early warning system based on multi-source data fusion, which is used to solve the situation where only the internal environment of the refrigerator may experience rapid temperature changes, and it is difficult to detect the above-mentioned complex situation inside the refrigerator through food images alone, which may lead to false alarms, resulting in inaccurate system early warning results, thereby causing a decline in food safety performance.

[0006] To achieve the above-mentioned purpose, the first aspect of the present application provides a smart home early warning system based on multi-source data fusion, comprising: a refrigerator setting module, a data acquisition module, a data analysis module, an intelligent early warning module and a database;

[0007] State recognition module: obtains several refrigerator loads within a set target period; analyzes the several refrigerator loads to obtain a refrigerator state representing the current refrigerator working state;

[0008] Data acquisition module: obtains the refrigerator load of the target refrigerator, as well as its external ambient temperature and internal image data;

[0009] Data analysis module: When the refrigerator state is stable, the refrigerator load is analyzed based on the ambient temperature to obtain the theoretical load required at the ambient temperature; based on the image data analysis, each food type inside the refrigerator is obtained and analyzed to obtain the theoretical food thermal energy used to represent the heat generated by each food type;

[0010] Intelligent early warning module: obtains refrigerator load and generates early warning signals based on refrigerator load, theoretical load and theoretical thermal energy;

[0011] The food images at the time when the food is put in and the food images at the current moment are obtained from the image data respectively; the food images at the time when the food is put in are compared and analyzed with the food images corresponding to the current moment to obtain the food similarity and the current food position.

[0012] Preferably, the refrigerator status indicating the current working status of the refrigerator is obtained by analyzing the loads of the refrigerators, including:

[0013] The load variance of the corresponding refrigerators is obtained by calculating the load of several refrigerators, and the load variance is compared with the variance threshold. When the load variance does not exceed the threshold, the current refrigerator working state is stable; when the load variance exceeds the threshold, the current refrigerator working state is abnormal; the variance threshold is obtained from the database.

[0014] Preferably, analyzing the refrigerator load based on the ambient temperature to obtain a theoretical load required at the ambient temperature includes:

[0015] Obtain the ambient temperature value and the temperature value set inside the refrigerator when the refrigerator is in a stable state;

[0016] The absolute value of the difference between the ambient temperature and the corresponding set temperature inside the refrigerator is recorded as the temperature difference;

[0017] The product of the temperature difference and the load required for unit temperature change is recorded as the theoretical load;

[0018] Preferably, the various types of food inside the refrigerator are obtained based on image data analysis, including:

[0019] The image data is imported into the foreign body recognition model to obtain several key features; the key features include the shape of the food, the texture of the food surface, etc. The extracted key features are compared with a pre-built food feature library to identify the types of food in the image.

[0020] Preferably, the foreign body recognition model is obtained by training an artificial intelligence model, including:

[0021] A plurality of image data and key features are obtained from a database; the key features are obtained by experts analyzing the plurality of image data; specifically, based on the characteristics of each food in the image data, the key features corresponding to each food are extracted, and the key features corresponding to each food are integrated into a food category; the plurality of image data and the corresponding key features are integrated into a plurality of training data and test data;

[0022] Import several training data into the artificial intelligence model for training, and test the trained artificial intelligence model with test data; specifically, input the image data in the test data into the trained artificial intelligence model, and output the key features. Compare the key features with the key features recorded in the test data to determine whether they are within the acceptable range; if yes, it means that the group of test data has passed the test, and continue to test the next group of test data; if not, it is necessary to adjust the relevant parameters of the artificial intelligence model, and continue to use the group of test data for testing; until a set proportion of test data passes the test; finally, the input is image data and key features, and the output is a foreign body recognition model; the artificial intelligence model is a CNN intelligent model, etc.

[0023] Preferably, the pre-built food feature library includes:

[0024] Obtain a large number of images corresponding to different types of food, annotate these images, determine the food types and specific feature parameters in the images, and build a food feature library;

[0025] Preferably, obtaining each of the food types and analyzing them to obtain theoretical food thermal energy for representing the heat generated by each type of food includes:

[0026] Obtain each food type, the corresponding food quantity, and the calories released by the corresponding individual food, and multiply the food quantity of the current food type and the calories released by the corresponding individual food to obtain the calories of the current food type;

[0027] The theoretical food energy is obtained by summing up the calories corresponding to each food type;

[0028] Preferably, generating a warning signal based on the refrigerator load, theoretical load, and theoretical food thermal energy includes:

[0029] The difference between the refrigerator load and the theoretical load is used to obtain the food load used to represent the total production capacity inside the refrigerator; the actual food heat energy is obtained by multiplying the food load and the converted heat per unit load;

[0030] Comparing the actual food thermal energy with the theoretical food thermal energy to generate a warning signal;

[0031] Preferably, comparing the actual food thermal energy with the theoretical food thermal energy to generate a warning signal includes:

[0032] When the actual food heat energy is greater than the theoretical food heat energy, an early warning signal is generated;

[0033] When the actual food heat energy is less than the theoretical food heat energy, a safety signal is generated;

[0034] Preferably, comparing and analyzing each food image at the time when the food is placed with each image corresponding to the current time to obtain the food similarity includes:

[0035] Obtaining images of each food at the time the food is put in and images of the food corresponding to the current time;

[0036] Obtaining a food similarity threshold for each food image from a database; the food similarity threshold is a threshold set when food deteriorates;

[0037] Compare and analyze the food image at the time of entry with the image corresponding to the current time to obtain the food similarity;

[0038] Each food image is judged in turn;

[0039] When the food similarity is less than the food similarity threshold, the current food position is obtained;

[0040] The current food location is the specific location inside the refrigerator where the food has deteriorated;

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1. This application obtains several refrigerator loads within a set target time period; analyzes the several refrigerator loads to obtain a refrigerator status representing the current refrigerator operating status; when the refrigerator status is stable, obtains the external ambient temperature and internal image data of the target refrigerator; analyzes the refrigerator load based on the ambient temperature to obtain a theoretical load required to represent the ambient temperature; obtains various food types inside the refrigerator based on image data analysis, obtains the theoretical food thermal energy used to represent the heat generated by each food type, obtains the refrigerator load, and generates an early warning signal based on the refrigerator load, theoretical load, and theoretical thermal energy. By analyzing the internal characteristics of the refrigerator for early warning processing, false alarms are greatly avoided, thereby improving the accuracy of the early warning results;

[0043] 2. This application obtains the food similarity by comparing and analyzing the food images at the time the food is placed with the images corresponding to the current moment, and determines whether the food similarity is within the food similarity threshold range; determines the specific location of the food by analyzing the similarity of the food; and greatly improves the efficiency of detecting food spoilage. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 This is a schematic diagram of the principles of this application;

[0046] Figure 2 This is a flow chart of the principles of this application. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions of this application in conjunction with the embodiments. Obviously, the embodiments described are only a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] See also Figure 1 , the first embodiment of the present application provides a smart home early warning system based on multi-source data fusion, including: a refrigerator setting module, a data acquisition module, a data analysis module, an intelligent early warning module and a database;

[0049] State recognition module: obtains several refrigerator loads within a set target period; analyzes the several refrigerator loads to obtain a refrigerator state representing the current refrigerator working state;

[0050] Data acquisition module: when the refrigerator state is stable, obtain the external ambient temperature and internal image data of the target refrigerator;

[0051] Data analysis module: Analyzes the refrigerator load based on the ambient temperature to obtain a theoretical load required at the ambient temperature; obtains various types of food inside the refrigerator based on image data analysis, obtains each of the food types and analyzes them to obtain theoretical food thermal energy to represent the heat generated by each type of food;

[0052] Intelligent early warning module: obtains refrigerator load and generates early warning signals based on refrigerator load, theoretical load and theoretical thermal energy;

[0053] The food images at the time when the food is put in and the food images at the current moment are obtained from the image data respectively; the food images at the time when the food is put in are compared and analyzed with the food images corresponding to the current moment to obtain the food similarity and the current food position.

[0054] Specifically, the refrigerator status representing the current working status of the refrigerator is obtained by analyzing the loads of the refrigerator, including:

[0055] The load variance of the corresponding refrigerators is obtained by calculating the load of several refrigerators, and the load variance is compared with the variance threshold. When the load variance does not exceed the threshold, the current refrigerator working state is stable; when the load variance exceeds the threshold, the current refrigerator working state is abnormal; the variance threshold is obtained from the database.

[0056] In this embodiment, it should be explained that the load variance of the corresponding refrigerator is obtained by calculating several load quantities. The smaller the variance is, the more stable the load required by the corresponding refrigerator is, and the more stable the working state of the corresponding refrigerator is.

[0057] The refrigerator load is the load required by the refrigerator to maintain the current working state; the refrigerator state is the current working state of the refrigerator; and the image data is an image of food obtained from inside the refrigerator.

[0058] Specifically, analyzing the refrigerator load based on the ambient temperature to obtain a theoretical load required at the ambient temperature includes:

[0059] Obtain the ambient temperature value and the temperature value set inside the refrigerator when the refrigerator is in a stable state;

[0060] The absolute value of the difference between the ambient temperature and the corresponding set temperature inside the refrigerator is recorded as the temperature difference;

[0061] The product of the temperature difference and the load required for unit temperature change is recorded as the theoretical load;

[0062] In this embodiment, it should be explained that the load required for unit temperature has taken into account the load required by factors such as the refrigerator material properties and area; the theoretical load is the load that the refrigerator needs to maintain at the external ambient temperature.

[0063] Specifically, the various types of food inside the refrigerator are obtained based on image data analysis, including:

[0064] The image data is imported into the foreign body recognition model to obtain several key features; the key features include the shape of the food, the texture of the food surface, etc. The extracted key features are compared with a pre-built food feature library to identify the types of food in the image.

[0065] Specifically, the foreign object recognition model is obtained through training of the artificial intelligence model, including:

[0066] A plurality of image data and key features are obtained from a database; the key features are obtained by experts analyzing the plurality of image data; specifically, based on the characteristics of each food in the image data, the key features corresponding to each food are extracted, and the key features corresponding to each food are integrated into a food category; the plurality of image data and the corresponding key features are integrated into a plurality of training data and test data;

[0067] Import several training data into the artificial intelligence model for training, and test the trained artificial intelligence model with test data; specifically, input the image data in the test data into the trained artificial intelligence model, and output the key features. Compare the key features with the key features recorded in the test data to determine whether they are within the acceptable range; if yes, it means that the test data has passed the test, and continue to test the next set of test data; if not, it is necessary to adjust the relevant parameters of the artificial intelligence model, and continue to use the test data for testing; until a set proportion of test data passes the test; finally, the input is image data and key features, and the output is a foreign body recognition model; the artificial intelligence model is a CNN intelligent model, etc.

[0068] The foreign body recognition model is an artificial intelligence model.

[0069] Specifically, the pre-built food feature library includes:

[0070] A large number of images corresponding to different types of food are obtained, these images are annotated, the types of food in the images and specific feature parameters are determined, and a food feature library is constructed; the food feature library is a database for storing the feature parameters corresponding to each food type.

[0071] Specifically, each of the food types is obtained and analyzed to obtain theoretical food thermal energy for representing the heat generated by each type of food, including:

[0072] Obtain each food type, the corresponding food quantity, and the calories released by the corresponding individual food, and multiply the food quantity of the current food type and the calories released by the corresponding individual food to obtain the calories of the current food type;

[0073] The theoretical food energy is obtained by summing up the calories corresponding to each food type;

[0074] In this embodiment, it needs to be explained that the more food there is in the current food type, the greater the heat released by the individual food corresponding to the current food type, which means that the oxidation of the corresponding food of this type inside the refrigerator is greater, and the corresponding heat of the current food type is higher, and vice versa; the theoretical food thermal energy is the heat theoretically generated by various types of food inside the refrigerator.

[0075] Specifically, the early warning signal is generated based on the refrigerator load, theoretical load, and theoretical food thermal energy, including:

[0076] The difference between the refrigerator load and the theoretical load is used to obtain the food load used to represent the total production capacity inside the refrigerator; the actual food heat energy is obtained by multiplying the food load and the converted heat per unit load;

[0077] Comparing the actual food thermal energy with the theoretical food thermal energy to generate a warning signal;

[0078] Specifically, comparing the actual food thermal energy with the theoretical food thermal energy to generate a warning signal includes:

[0079] When the actual food heat energy is greater than the theoretical food heat energy, an early warning signal is generated;

[0080] When the actual food heat energy is less than the theoretical food heat energy, a safety signal is generated;

[0081] In this embodiment, it should be explained that when the actual food thermal energy is greater than the theoretical food thermal energy, it indicates that the probability of oxidation of the food inside the refrigerator increases, which in turn causes the food to spoil; the system will issue an early warning and remind the user to take timely action; when the actual food thermal energy is less than the theoretical food thermal energy, it indicates that the food inside the refrigerator is fresh; the early warning signal is a warning issued when spoiled food appears in the refrigerator; the safety signal indicates that the food in the refrigerator is safe; the actual food thermal energy is the actual amount of heat generated by various types of food inside the refrigerator;

[0082] Specifically, comparing and analyzing each food image at the time when the food is placed with each image corresponding to the current time to obtain the food similarity includes:

[0083] Obtaining images of each food at the time when the food is put in and images of each food corresponding to the current time;

[0084] Obtaining a food similarity threshold for each food image from a database; the food similarity threshold is a threshold set when food deteriorates;

[0085] Compare and analyze the food image at the time of entry with the food image corresponding to the current time to obtain the food similarity;

[0086] Each food image is judged in turn;

[0087] Determine whether the food similarity is within the food similarity threshold; if yes, obtain the current food location;

[0088] The current food location is the specific location inside the refrigerator where the food has deteriorated;

[0089] In this embodiment, it should be explained that the food similarity is obtained by comparing and analyzing the food image at the time of placement with the image corresponding to the current moment, and then determining whether the food similarity is within a food similarity threshold. If so, the probability of the current food being damaged is greater, and then obtaining the specific location of the food corresponding to the current image. The food images at the time of placement are the food images captured at the time of placement, and the food images at the current moment are the food images captured at the current moment. The food similarity is the similarity between the food images at the time of placement and the current moment.

[0090] The present application compares and analyzes the food images at the time the food was placed with the images corresponding to the current time to obtain the food similarity, and determines whether the food similarity is within the food similarity threshold. The specific location of the food is determined by analyzing the food similarity, which greatly improves the efficiency of detecting food spoilage.

[0091] See also Figure 2 , the second embodiment of the present application provides a principle flow chart;

[0092] Some of the data in the above formula are calculated by removing the dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0093] The working principle of this application is as follows: obtaining several refrigerator loads within a set target time period; analyzing these refrigerator loads to obtain a refrigerator status representing the current refrigerator operating status; when the refrigerator status is stable, obtaining the external ambient temperature and internal image data of the target refrigerator; analyzing the refrigerator load based on the ambient temperature to obtain a theoretical load representing the required load at the ambient temperature; analyzing the image data to obtain the various food types inside the refrigerator, obtaining the theoretical food thermal energy representing the heat generated by each food type; obtaining the refrigerator load, and generating an early warning signal based on the refrigerator load, theoretical load, and theoretical thermal energy. By analyzing the internal characteristics of the refrigerator for early warning processing, false alarms are greatly avoided, thereby improving the accuracy of the early warning results.

[0094] The above embodiments are only used to illustrate the technical method of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present application.

Claims

1. The smart home early warning system based on multi-source data fusion is characterized by: State identification module: obtains several refrigerator loads within a set target period; analyzes the several refrigerator loads to obtain a refrigerator state representing the current refrigerator working state; the refrigerator state includes stable and abnormal; Data acquisition module: when the refrigerator state is stable, obtain the external ambient temperature and internal image data of the target refrigerator; Data analysis module: Analyzes the refrigerator load based on the ambient temperature to obtain a theoretical load required at the ambient temperature; obtains various types of food inside the refrigerator based on image data analysis, obtains each of the food types and analyzes them to obtain theoretical food thermal energy to represent the heat generated by each type of food; Intelligent early warning module: obtains refrigerator load and generates early warning signals based on refrigerator load, theoretical load and theoretical thermal energy.

2. The smart home early warning system based on multi-source data fusion according to claim 1 is characterized in that: The method for obtaining the refrigerator status includes: The load variance of the corresponding refrigerators is calculated by calculating the load of several refrigerators, and the load variance is compared with the preset variance threshold. When the load variance does not exceed the threshold, the current refrigerator working state is recorded as a stable state; when the load variance exceeds the threshold, the current refrigerator working state is recorded as an abnormal state.

3. The smart home early warning system based on multi-source data fusion according to claim 1 is characterized in that: Analyzing the refrigerator load based on the ambient temperature to obtain the theoretical load required at the ambient temperature includes: Obtain the ambient temperature value and the temperature value set inside the refrigerator when the refrigerator is in a stable state; The absolute value of the difference between the ambient temperature and the corresponding temperature set inside the refrigerator is recorded as the temperature difference; The product of the temperature difference and the load required for unit temperature change is recorded as the theoretical load.

4. The smart home early warning system based on multi-source data fusion according to claim 1 is characterized in that: The various food types inside the refrigerator obtained based on image data analysis include: The image data is imported into the foreign body recognition model to obtain several key features; the key features include the shape of the food, the texture of the food surface, etc. The extracted key features are compared with a pre-built food feature library to identify the types of food in the image.

5. The smart home early warning system based on multi-source data fusion according to claim 4 is characterized in that: The foreign body recognition model is obtained through training of an artificial intelligence model, including: Acquire a number of image data and key features from a database; the key features are obtained by analyzing the number of image data by experts; and the number of image data and corresponding key features are integrated into a number of training data and test data; Import several training data into the artificial intelligence model for training, and test the trained artificial intelligence model through test data; the final input is image data and key features, and the output is a foreign object recognition model.

6. The smart home early warning system based on multi-source data fusion according to claim 4 is characterized in that: The pre-built food feature library includes: Obtain a large number of images corresponding to different types of food, annotate these images, determine the food types and specific feature parameters in the images, and build a food feature library.

7. The smart home early warning system based on multi-source data fusion according to claim 1 is characterized in that: The theoretical food energy used to represent the heat generated by each type of food is obtained by analyzing each type of food, including: Obtain each food type, the corresponding food quantity, and the calories released by the corresponding individual food, and multiply the food quantity of the current food type and the calories released by the corresponding individual food to obtain the calories of the current food type; The theoretical food energy is obtained by summing up the calories corresponding to each food type.

8. The smart home early warning system based on multi-source data fusion according to claim 1 is characterized in that: The generating of the warning signal based on the refrigerator load, theoretical load and theoretical food thermal energy includes: The difference between the refrigerator load and the theoretical load is used to obtain the food load used to represent the total production capacity inside the refrigerator; the actual food heat energy is obtained by multiplying the food load and the converted heat per unit load; The actual food thermal energy is compared with the theoretical food thermal energy to generate a warning signal.

9. The smart home early warning system based on multi-source data fusion according to claim 1, characterized in that: Comparing the actual food heat energy with the theoretical food heat energy to generate the warning signal includes: When the actual food heat energy is greater than the theoretical food heat energy, an early warning signal is generated; When the actual food thermal energy is less than the theoretical food thermal energy, a safety signal is generated.

10. The smart home early warning system based on multi-source data fusion according to claim 1, characterized in that: The image data is also used to determine abnormal food, including: Obtain food images at the time when each food is put in, and extract the image of each food in the image data; Comparing and analyzing each food image at the time when the food is placed with each food image corresponding to the current time to obtain the food similarity, further comprising: Get the images corresponding to the food at the current moment; Obtaining a food similarity threshold for each food image from a database; the food similarity threshold is a threshold set when food deteriorates; Compare and analyze the food image at the time of entry with the food image corresponding to the current time to obtain the food similarity; Each food image is judged in turn; When the food similarity is less than the food similarity threshold, the current food position is obtained.

Images