Method, device and equipment for detecting food in refrigerator, medium and product

Abstract

The embodiment of the invention provides a method, device and equipment for detecting food in a refrigerator, a medium and a product. The method comprises the following steps: acquiring current image data of the food in the refrigerator; key feature information of food is determined from the current image data; determining a plurality of maturity evaluation indexes according to the key feature information; and generating a food maturity evaluation result according to the plurality of maturity evaluation indexes. According to the embodiment of the invention, the current image data of the food in the refrigerator is obtained, and the data is further processed to evaluate the maturity of the food, so that errors caused by manual judgment of the maturity of the food are avoided, and food waste can be reduced while the life quality of a user is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of smart home appliances, and in particular to a method, device, equipment, medium and product for detecting food in a refrigerator. Background Art

[0002] As an indispensable household appliance in modern life, the main function of a refrigerator is to provide users with a low-temperature storage environment to extend the shelf life of food.

[0003] However, in order to reduce losses during transportation, some merchants pick, transport and sell most foods (such as fruits and vegetables) when they are immature. When users buy food, they manually analyze the food to determine whether it is mature. However, due to misjudgment of the food during the purchase process or keeping it in the refrigerator for too long, the eating effect is often unsatisfactory, such as the food is immature or the inside is already corrupt, which affects the user experience. Summary of the invention

[0004] In view of the above problems, a method, device, equipment, medium and product for detecting food in a refrigerator are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A method for detecting food in a refrigerator, the method comprising:

[0006] Get the current image data of the food in the refrigerator;

[0007] Determining key feature information of the food from the current image data;

[0008] Determining a plurality of maturity assessment indicators based on the key characteristic information;

[0009] A maturity assessment result of the food is generated according to the multiple maturity assessment indicators.

[0010] Optionally, it also includes:

[0011] A recommended processing solution for the food is generated according to the maturity assessment result, and the maturity assessment result and the recommended processing solution for the food are displayed through the user interaction interface of the refrigerator.

[0012] Optionally, generating a maturity assessment result of the food according to the multiple maturity assessment indicators includes:

[0013] Determining weight information of the plurality of maturity assessment indicators;

[0014] According to the weight information, the plurality of maturity assessment indicators are weighted and summed to obtain a maturity assessment result of the food.

[0015] Optionally, the maturity assessment index includes any one of the following: color saturation, fruit shape, surface smoothness.

[0016] Optionally, before determining the key feature information of the food from the current image data, the method further includes:

[0017] Performing defogging processing on the current image data.

[0018] Optionally, before determining the key feature information of the food from the current image data, the method further includes:

[0019] The current image data is normalized.

[0020] Optionally, determining the key feature information of the food from the current image data includes:

[0021] The current image data is input into a pre-trained neural network model, and target detection is performed on the current image data through the neural network model to obtain key feature information of the food.

[0022] Optionally, the key feature information includes any one or more of the following: color information, shape information, and texture information.

[0023] Optionally, before acquiring the current image data of the food in the refrigerator, the method further includes:

[0024] Acquire sample image data, wherein the sample image data includes image data of different lighting scenes;

[0025] The sample image data is used to train a preset neural network model.

[0026] Optionally, before using the sample image data set to train a preset neural network model, the method further includes:

[0027] Performing defogging and / or normalization processing on the sample image data;

[0028] Before using the sample image data set to train the preset neural network model, the method further includes: expanding the sample image data.

[0029] A device for detecting food in a refrigerator, the device comprising:

[0030] A current image data acquisition module, used to acquire current image data of food in the refrigerator;

[0031] A key feature information determination module, used to determine the key feature information of the food from the current image data;

[0032] A maturity assessment indicator determination module, used to determine a plurality of maturity assessment indicators according to the key feature information;

[0033] The maturity assessment result generating module is used to generate the maturity assessment result of the food according to the multiple maturity assessment indicators.

[0034] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method described above when executed by the processor.

[0035] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0036] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described above.

[0037] The embodiments of the present invention have the following advantages:

[0038] In an embodiment of the present invention, by acquiring current image data of food in a refrigerator; determining key feature information of the food from the current image data; determining multiple maturity assessment indicators based on the key feature information; and generating a maturity assessment result of the food based on the multiple maturity assessment indicators, it is achieved that by acquiring current image data of food in a refrigerator and further processing the data to assess the maturity of the food, errors in human judgment of food maturity are avoided, the quality of life of users can be improved and food waste can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0040] Figure 1 is a flowchart of steps of a method for detecting food in a refrigerator provided by some embodiments of the present invention;

[0041] Figure 2 is a flow chart of a method for detecting food in a refrigerator provided by some embodiments of the present invention;

[0042] Figure 3 is a flowchart of steps of another method for detecting food in a refrigerator provided by some embodiments of the present invention;

[0043] Figure 4 It is a structural block diagram of a device for detecting food in a refrigerator provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In practical applications, the maturity of food (such as fruits and vegetables) affects consumers' eating experience and health. Specifically, unripe fruits and vegetables often have a poor taste and may be irregular or odd in shape. More importantly, they may contain substances that are harmful to the human body, such as nitrites, which may pose a threat to health when consumed in excess. On the contrary, although overripe fruits and vegetables may taste better, their storage difficulty increases significantly, which can easily lead to the rapid loss of nutrients. At the same time, excessive maturity also makes fruits and vegetables more susceptible to microbial invasion, causing decay and deterioration, further shortening the edible period.

[0046] In related technologies, methods for judging the maturity of fruits and vegetables mostly rely on manual experience, such as observing the color, texture, shape, etc. of fruits and vegetables with the naked eye; this is not only time-consuming and labor-intensive, but the test results are easily affected by human factors.

[0047] In an embodiment of the present invention, an improved Fast R-CNN algorithm (Fast Region-based Convolutional Neural Networks, target detection algorithm) is proposed to detect the maturity of fruits and vegetables in a refrigerator. The images of fruits and vegetables are defogged to enhance the image features of the fruits and vegetables. Then, the fruit and vegetable image data are input into the Fast R-CNN network, and the network introduces an attention mechanism to adaptively improve the model's attention to the key features of fruits and vegetables. Then, according to the key features of fruits and vegetables, a set of scientific and reasonable maturity evaluation indicators is established to comprehensively judge the maturity status of fruits and vegetables. Finally, the maturity information of fruits and vegetables is fed back to the user through a user interaction interface, and the user selectively processes fruits and vegetables of various maturity status according to the feedback information.

[0048] Through the fruit and vegetable maturity detection method based on the improved Fast R-CNN in the embodiment of the present invention, it is possible to reduce labor costs, break through the dependence of traditional detection technology on the detection environment and the experience of detection personnel, and effectively reduce the detection cost of industrial-grade detection equipment; and the solution can be equipped with an automated intelligent detection function, can adapt to the complex environmental conditions in the refrigerator, improve the accuracy of the fruit and vegetable maturity detection results, and provide users with reliable fruit and vegetable maturity prompts.

[0049] The present invention will be further described below in conjunction with the accompanying drawings:

[0050] Reference Figure 1 , shows a flowchart of a method for detecting food in a refrigerator provided by some embodiments of the present invention, which may specifically include the following steps:

[0051] Step 101, obtaining current image data of food in a refrigerator.

[0052] Among them, food can be fruits and vegetables (fruits and vegetables), such as apples, bananas, oranges, spinach, lettuce, tomatoes, etc.

[0053] As some examples, a suitable image acquisition device, such as a high-definition camera, can be installed inside or outside the refrigerator (through a perspective window); the device can capture the situation inside the refrigerator and also has the function of automatically adjusting the aperture and exposure time to adapt to the lighting changes in the refrigerator.

[0054] In some embodiments, once the image acquisition device is installed and connected to the network, it can begin to capture images of the interior of the refrigerator periodically or in real time, where the images can be static photos or dynamic video frames, for example, once every hour or every time the refrigerator door is opened.

[0055] For example, the cloud server can control the camera to periodically capture images of food in the refrigerator to obtain current image data of the food in the refrigerator. The image data captured by the camera can be transmitted to the cloud server via the network.

[0056] In some embodiments of the present invention, before obtaining the current image data of the food in the refrigerator, the method further includes: obtaining sample image data, wherein the sample image data includes image data of different lighting scenes; and using the sample image data to train a preset neural network model.

[0057] The sample image data is a series of images used to train or test the neural network model.

[0058] As some examples, image data of different foods, foods of different maturity, and foods under different lighting scenes may be collected in advance by a camera and used as sample image data.

[0059] In actual applications, there will be changes in lighting inside the refrigerator due to various reasons, such as aging of the bulb inside the refrigerator, loose bulbs, low ambient temperature, dust, opening and closing the refrigerator door, etc.; by adding image data of different lighting scenes to the sample data to train the preset neural network model, the ability of the neural network model to detect food targets can be improved.

[0060] In some examples, a sample data set can also be established based on the sample image data. The sample data set can be labeled to distinguish fruits and vegetables of different maturity levels. To enhance the robustness and generalization ability of the model, the data set can select two data sets under normal light and low light.

[0061] As some examples, the preset neural network model can be a target detection model (Fast Region-based Convolutional Neural Networks, Fast R-CNN); after acquiring sample image data, the sample image data can be input into the neural network model, the neural network model can be trained, and the parameters in the model can be adjusted to minimize the difference between the predicted results and the actual results.

[0062] In some examples, the object detection model (Fast R-CNN) is a deep learning algorithm for object detection, which includes:

[0063] 1. Use the Selective Search algorithm to extract candidate regions of the image. For example, use Selective Search to segment the image and automatically generate candidate regions. These regions can contain labeled categories and map the candidate regions to the convolutional feature layer.

[0064] 2. Normalize the convolutional feature layer.

[0065] 3. Use Softmax for classification, regress the position of the candidate area, and use regression analysis to fine-tune the bounding box position of the candidate area to more accurately locate the target object.

[0066] For example, suppose there is an image that contains an apple. Using the Selective Search algorithm, multiple candidate regions can be obtained, some of which contain apples and some do not. Through Softmax classification, the probability that each candidate region belongs to the "apple" category can be obtained. Then, for the candidate region classified as "apple", bounding box regression is used to adjust its position so that it more closely surrounds the outline of the "apple".

[0067] For example, if the coordinates of the original candidate region are (x1, y1, x2, y2), the regression analysis can output an offset (Δx, Δy, Δw, Δh), which is applied to the original coordinates to obtain new and more accurate bounding box coordinates (x1+Δx, y1+Δy, x2+Δw, y2+Δh).

[0068] In some embodiments of the present invention, before using the sample image data set to train a preset neural network model, it also includes: performing defogging and / or normalization processing on the sample image data; and performing expansion processing on the sample image data.

[0069] Before using the sample image data set to train the preset neural network model, the sample data can be preprocessed through a preprocessing module, such as defogging, normalization, expansion, etc.

[0070] Among them, defog processing can be an image enhancement technology for fog or blur in image data; in the actual environment inside the refrigerator, due to the cold air inside the refrigerator, the image may appear blurred, the contrast is reduced, and the color is unsaturated; defog processing can reduce or eliminate these effects, making the image clearer, the contrast is higher, and the colors are more vivid.

[0071] Normalization processing may include resizing and color standardization; resizing is to ensure that the image size can meet the requirements of model training. Some models have requirements for the size of input data (image), and the image sizes obtained by taking photos may be inconsistent. In this case, the size needs to be adjusted. Resizing helps to improve the model's image processing speed and ensure that the neural network model can maintain a consistent input data format when processing different images, thereby improving the stability and accuracy of the model; color standardization is to convert the pixel values ​​of the image (such as integers between 0 and 255) into a standardized range (such as decimals between 0 and 1). This processing can make the image data more evenly distributed and help the model converge faster.

[0072] The expansion processing can be to enhance the image to increase the data volume of the model; such as rotating, cropping, changing the color, adding noise and other changes to the image data to improve the generalization ability of the model, prevent the subsequent model from overfitting, and improve the accuracy of the model's detection results of fruit and vegetable maturity.

[0073] Step 102: determining key feature information of the food from the current image data.

[0074] In some embodiments of the present invention, the key feature information includes any one or more of the following: color information, shape information, and texture information.

[0075] In practical applications, image processing technology can be used to process the current image data to determine the color information, shape information, and texture information of the food in the image. For example, when processing an image of a fruit, the shape, color, and texture of the fruit can be identified first, and then through comparison and analysis, the type of fruit can be determined, as well as its freshness and maturity.

[0076] As some examples, after the current image data is captured by the camera, it can be transmitted to a cloud server through the network. The cloud server can pre-process the current image data and input it into a pre-trained neural network model after pre-processing. The neural network model can be used to perform target detection on the current image data to output key feature information of the food.

[0077] In some embodiments of the present invention, determining the key feature information of the food from the current image data includes: inputting the current image data into a pre-trained neural network model, performing target detection on the current image data through the neural network model, and obtaining the key feature information of the food.

[0078] Among them, target detection can identify foods in images and determine their positions and ranges in the images. Target detection can extract information from image data that can represent attributes of food, such as food category, shape, color, texture, etc., and use it as key feature information of food.

[0079] For example, the current image data can be input into a pre-trained target detection model, and the target detection model is used to perform target detection on the current image data to obtain color information, shape information, texture information, etc. of the food.

[0080] In some embodiments of the present invention, before determining the key feature information of the food from the current image data, the method further includes: performing a defogging process on the current image data.

[0081] In some embodiments of the present invention, before determining the key feature information of the food from the current image data, the method further includes: normalizing the current image data.

[0082] As some examples, the current image data may be preprocessed by a preset preprocessing module, such as defogging, normalization, expansion, etc.

[0083] Among them, dehazing processing can apply image dehazing algorithms such as DehazeNet (an image dehazing algorithm based on deep learning) to eliminate fog interference caused by environmental humidity and enhance image clarity and contrast; dehazing processing can include: feature extraction, multi-scale mapping, local extrema and nonlinear regression.

[0084] In the specific implementation, the features of the image are first extracted through the convolution layer. These features include texture, color and brightness information in the foggy image. After feature extraction, DehazeNet uses convolution kernels of different scales to capture multi-scale information in the image. The algorithm further refines the estimate of the transmittance map by finding local extrema in the image. Local extrema usually correspond to edge and texture information in the image, which is crucial for restoring the details and structure of the image. Finally, DehazeNet uses nonlinear regression layers, such as the BReLU activation function, to adjust the transmittance map and the restored image, and further optimizes the transmittance map through nonlinear transformation, so that the restored image is closer to the real fog-free image in color and brightness.

[0085] After dehazing the current image data, the dehazed image can be normalized, such as resizing and color standardization, to ensure the consistency of the image before inputting into the pre-trained neural network model.

[0086] Normalization processing may include resizing and color standardization; resizing is to ensure that the image size can meet the requirements of model training. Some models have requirements for the size of input data (image), and the image sizes obtained by taking photos may be inconsistent. In this case, the size needs to be adjusted. Resizing helps to improve the model's image processing speed and ensure that the neural network model can maintain a consistent input data format when processing different images, thereby improving the stability and accuracy of the model; color standardization is to convert the pixel values ​​of the image (such as integers between 0 and 255) into a standardized range (such as decimals between 0 and 1). This processing can make the image data more evenly distributed and help the model converge faster.

[0087] After normalizing the current image data, in order to increase the amount of image data, the image can be enhanced to prevent overfitting of the subsequent model and improve the accuracy of the model's detection results of fruit and vegetable maturity.

[0088] The expansion processing can be to enhance the image to increase the data volume of the model; such as rotating, cropping, changing the color, adding noise and other changes to the image data to improve the generalization ability of the model, prevent the subsequent model from overfitting, and improve the accuracy of the model's detection results of fruit and vegetable maturity.

[0089] Step 103: Determine a plurality of maturity evaluation indicators based on the key feature information.

[0090] In some embodiments of the present invention, the maturity assessment index includes any one of the following: color saturation, fruit shape, and surface smoothness.

[0091] After obtaining the key feature information, the color information, shape information, and texture information of the food can be determined. By analyzing it with multiple maturity assessment indicators (such as color saturation, fruit shape, and surface smoothness), the maturity of the food can be determined.

[0092] Step 104: Generate a maturity assessment result of the food according to the plurality of maturity assessment indicators.

[0093] As some examples, a food maturity assessment result may be generated through a food maturity assessment module.

[0094] For example, the food maturity assessment module can extract key feature information output after processing by the neural network model, input the key feature information into the food maturity assessment module, and comprehensively calculate the maturity assessment results (such as maturity score) of each food through multiple maturity assessment indicators (such as color saturation, fruit shape, surface smoothness, etc.).

[0095] In some embodiments of the present invention, generating the maturity assessment result of the food based on the multiple maturity assessment indicators includes: determining weight information of the multiple maturity assessment indicators; and performing weighted summation of the multiple maturity assessment indicators according to the weight information to obtain the maturity assessment result of the food.

[0096] Among them, the weight information of the maturity assessment index can be set with different weights according to the degree of influence on maturity, such as the weight of color saturation is 1, the weight of fruit shape is 2, and the weight of surface smoothness is 3. The weighted sum of these three maturity assessment indexes is performed to obtain the maturity assessment result of the food:

[0097] Food maturity assessment result = x1 color saturation + x2 fruit shape + x3 surface smoothness.

[0098] In some embodiments of the present invention, the method further includes: generating a recommended processing solution for the food according to the maturity assessment result, and displaying the maturity assessment result and the recommended processing solution for the food through the user interaction interface of the refrigerator.

[0099] In actual applications, the user interaction interface can display the food maturity assessment results in real time through a high-definition display screen, and provide recommended food processing solutions (such as eating suggestions); secondly, the user interaction interface is also equipped with historical detection records to better manage the storage and consumption of fruits and vegetables in the refrigerator.

[0100] For example, the apples purchased on October 11 are overripe and their taste may be affected, so please eat them within two days. The cucumbers purchased on October 11 are spoiled and their effects on your health may be affected, so please do not eat them.

[0101] As some examples, the user interface may also display the food maturity assessment results and recommended treatment plans in a table, such as Table 1:

[0102]

[0103] Table 1

[0104] In some examples, the purchase time and the storage time can be determined based on the image data acquired by the camera, and the user can also adjust the purchase time and the storage time by himself through the user interaction interface.

[0105] In an embodiment of the present invention, by acquiring current image data of food in a refrigerator; determining key feature information of the food from the current image data; determining multiple maturity assessment indicators based on the key feature information; and generating a maturity assessment result of the food based on the multiple maturity assessment indicators, it is achieved that by acquiring current image data of food in a refrigerator and further processing the data to assess the maturity of the food, errors in human judgment of food maturity are avoided, the quality of life of users can be improved and food waste can be reduced.

[0106] The following is combined with Figure 2 The present invention is exemplified as follows:

[0107] Step 201: food image collection.

[0108] For example, the cloud server can control the camera to periodically capture images of food in the refrigerator to obtain image data of the food in the refrigerator. The image data captured by the camera can be transmitted to the cloud server through the network.

[0109] In step 202, the cloud server pre-processes the food image, establishes the food maturity model, improves the food maturity model, and evaluates the food maturity model; wherein the food image pre-processing may include: food image defogging processing, food image normalization processing, and food image enhancement processing.

[0110] The image data captured by the camera is used as sample image data to train the preset neural network model to obtain a food maturity model (trained target detection model), and the maturity of the food in the image data is evaluated through the food maturity evaluation module built into the food maturity model.

[0111] Step 203: user interaction interface.

[0112] Step 204: outputting the food maturity assessment result.

[0113] According to the maturity assessment results, a recommended treatment plan for the food is generated, and the maturity assessment results and the recommended treatment plan for the food are displayed through the user interaction interface of the refrigerator.

[0114] Reference Figure 3 , shows a flowchart of another method for detecting food in a refrigerator provided by some embodiments of the present invention, which may specifically include the following steps:

[0115] Step 301, obtaining current image data of food in the refrigerator.

[0116] Among them, food can be fruits and vegetables (fruits and vegetables), such as apples, bananas, oranges, spinach, lettuce, tomatoes, etc.

[0117] As some examples, a suitable image acquisition device, such as a high-definition camera, can be installed inside or outside the refrigerator (through a perspective window); the device can capture the situation inside the refrigerator and also has the function of automatically adjusting the aperture and exposure time to adapt to the lighting changes in the refrigerator.

[0118] In some embodiments, once the image acquisition device is installed and connected to the network, it can begin to capture images of the interior of the refrigerator periodically or in real time, where the images can be static photos or dynamic video frames, for example, once every hour or every time the refrigerator door is opened.

[0119] For example, the cloud server can control the camera to periodically capture images of food in the refrigerator to obtain current image data of the food in the refrigerator. The image data captured by the camera can be transmitted to the cloud server via the network.

[0120] In some embodiments of the present invention, before obtaining the current image data of the food in the refrigerator, the method further includes: obtaining sample image data, wherein the sample image data includes image data of different lighting scenes; and using the sample image data to train a preset neural network model.

[0121] The sample image data is a series of images used to train or test the neural network model.

[0122] As some examples, image data of different foods, foods of different maturity, and foods under different lighting scenes may be collected in advance by a camera and used as sample image data.

[0123] In actual applications, there will be changes in lighting inside the refrigerator due to various reasons, such as aging of the bulb inside the refrigerator, loose bulbs, low ambient temperature, dust, opening and closing the refrigerator door, etc.; by adding image data of different lighting scenes to the sample data to train the preset neural network model, the ability of the neural network model to detect food targets can be improved.

[0124] In some examples, a sample data set can also be established based on the sample image data. The sample data set can be labeled to distinguish fruits and vegetables of different maturity levels. To enhance the robustness and generalization ability of the model, the data set can select two data sets under normal light and low light.

[0125] As some examples, the preset neural network model can be a target detection model (Fast Region-based Convolutional Neural Networks, Fast R-CNN); after acquiring sample image data, the sample image data can be input into the neural network model, the neural network model can be trained, and the parameters in the model can be adjusted to minimize the difference between the predicted results and the actual results.

[0126] In some examples, the object detection model (Fast R-CNN) is a deep learning algorithm for object detection, which includes:

[0127] The Selective Search algorithm is used to extract candidate regions of the image. For example, the image is segmented using Selective Search to automatically generate candidate regions, which can contain labeled categories, and the candidate regions are mapped to the convolutional feature layer.

[0128] Normalize the convolutional feature layer.

[0129] Softmax is used for classification, the position of the candidate area is regressed, and regression analysis is used to fine-tune the bounding box position of the candidate area to more accurately locate the target object.

[0130] For example, suppose there is an image that contains an apple. Using the Selective Search algorithm, multiple candidate regions can be obtained, some of which contain apples and some do not. Through Softmax classification, the probability that each candidate region belongs to the "apple" category can be obtained. Then, for the candidate region classified as "apple", bounding box regression is used to adjust its position so that it more closely surrounds the outline of the "apple".

[0131] For example, if the coordinates of the original candidate region are (x1, y1, x2, y2), the regression analysis can output an offset (Δx, Δy, Δw, Δh), which is applied to the original coordinates to obtain new and more accurate bounding box coordinates (x1+Δx, y1+Δy, x2+Δw, y2+Δh).

[0132] In some embodiments of the present invention, before using the sample image data set to train a preset neural network model, it also includes: performing defogging and / or normalization processing on the sample image data; and performing expansion processing on the sample image data.

[0133] Before using the sample image data set to train the preset neural network model, the sample data can be preprocessed through a preprocessing module, such as defogging, normalization, expansion, etc.

[0134] Among them, defog processing can be an image enhancement technology for fog or blur in image data; in the actual environment inside the refrigerator, due to the cold air inside the refrigerator, the image may appear blurred, the contrast is reduced, and the color is unsaturated; defog processing can reduce or eliminate these effects, making the image clearer, the contrast is higher, and the colors are more vivid.

[0135] Normalization processing may include resizing and color standardization; resizing is to ensure that the image size can meet the requirements of model training. Some models have requirements for the size of input data (image), and the image sizes obtained by taking photos may be inconsistent. In this case, the size needs to be adjusted. Resizing helps to improve the model's image processing speed and ensure that the neural network model can maintain a consistent input data format when processing different images, thereby improving the stability and accuracy of the model; color standardization is to convert the pixel values ​​of the image (such as integers between 0 and 255) into a standardized range (such as decimals between 0 and 1). This processing can make the image data more evenly distributed and help the model converge faster.

[0136] The expansion processing can be to enhance the image to increase the data volume of the model; such as rotating, cropping, changing the color, adding noise and other changes to the image data to improve the generalization ability of the model, prevent the subsequent model from overfitting, and improve the accuracy of the model's detection results of fruit and vegetable maturity.

[0137] Step 302: determining key feature information of the food from the current image data.

[0138] In some embodiments of the present invention, the key feature information includes any one or more of the following: color information, shape information, and texture information.

[0139] In practical applications, image processing technology can be used to process the current image data to determine the color information, shape information, and texture information of the food in the image. For example, when processing an image of a fruit, the shape, color, and texture of the fruit can be identified first, and then through comparison and analysis, the type of fruit can be determined, as well as its freshness and maturity.

[0140] As some examples, after the current image data is captured by the camera, it can be transmitted to a cloud server through the network. The cloud server can pre-process the current image data and input it into a pre-trained neural network model after pre-processing. The neural network model can be used to perform target detection on the current image data to output key feature information of the food.

[0141] In some embodiments of the present invention, determining the key feature information of the food from the current image data includes: inputting the current image data into a pre-trained neural network model, performing target detection on the current image data through the neural network model, and obtaining the key feature information of the food.

[0142] Among them, target detection can identify foods in images and determine their positions and ranges in the images. Target detection can extract information from image data that can represent attributes of food, such as food category, shape, color, texture, etc., and use it as key feature information of food.

[0143] For example, the current image data can be input into a pre-trained target detection model, and the target detection model is used to perform target detection on the current image data to obtain color information, shape information, texture information, etc. of the food.

[0144] In some embodiments of the present invention, before determining the key feature information of the food from the current image data, the method further includes: performing a defogging process on the current image data.

[0145] In some embodiments of the present invention, before determining the key feature information of the food from the current image data, the method further includes: normalizing the current image data.

[0146] As some examples, the current image data may be preprocessed by a preset preprocessing module, such as defogging, normalization, expansion, etc.

[0147] Among them, dehazing processing can apply image dehazing algorithms such as DehazeNet (an image dehazing algorithm based on deep learning) to eliminate fog interference caused by environmental humidity and enhance image clarity and contrast; dehazing processing can include: feature extraction, multi-scale mapping, local extrema and nonlinear regression.

[0148] In the specific implementation, the features of the image are first extracted through the convolution layer. These features include texture, color and brightness information in the foggy image. After feature extraction, DehazeNet uses convolution kernels of different scales to capture multi-scale information in the image. The algorithm further refines the estimate of the transmittance map by finding local extrema in the image. Local extrema usually correspond to edge and texture information in the image, which is crucial for restoring the details and structure of the image. Finally, DehazeNet uses nonlinear regression layers, such as the BReLU activation function, to adjust the transmittance map and the restored image, and further optimizes the transmittance map through nonlinear transformation, so that the restored image is closer to the real fog-free image in color and brightness.

[0149] After dehazing the current image data, the dehazed image can be normalized, such as resizing and color standardization, to ensure the consistency of the image before inputting into the pre-trained neural network model.

[0150] Normalization processing may include resizing and color standardization; resizing is to ensure that the image size can meet the requirements of model training. Some models have requirements for the size of input data (image), and the image sizes obtained by taking photos may be inconsistent. In this case, the size needs to be adjusted. Resizing helps to improve the model's image processing speed and ensure that the neural network model can maintain a consistent input data format when processing different images, thereby improving the stability and accuracy of the model; color standardization is to convert the pixel values ​​of the image (such as integers between 0 and 255) into a standardized range (such as decimals between 0 and 1). This processing can make the image data more evenly distributed and help the model converge faster.

[0151] After normalizing the current image data, in order to increase the amount of image data, the image can be enhanced to prevent overfitting of the subsequent model and improve the accuracy of the model's detection results of fruit and vegetable maturity.

[0152] The expansion processing can be to enhance the image to increase the data volume of the model; such as rotating, cropping, changing the color, adding noise and other changes to the image data to improve the generalization ability of the model, prevent the subsequent model from overfitting, and improve the accuracy of the model's detection results of fruit and vegetable maturity.

[0153] Step 303: Determine a plurality of maturity evaluation indicators based on the key feature information.

[0154] In some embodiments of the present invention, the maturity assessment index includes any one of the following: color saturation, fruit shape, and surface smoothness.

[0155] After obtaining the key feature information, the color information, shape information, and texture information of the food can be determined. By analyzing it with multiple maturity assessment indicators (such as color saturation, fruit shape, and surface smoothness), the maturity of the food can be determined.

[0156] Step 304: Generate a maturity assessment result of the food according to the multiple maturity assessment indicators.

[0157] As some examples, a food maturity assessment result may be generated through a food maturity assessment module.

[0158] For example, the food maturity assessment module can extract key feature information output after processing by the neural network model, input the key feature information into the food maturity assessment module, and comprehensively calculate the maturity assessment results (such as maturity score) of each food through multiple maturity assessment indicators (such as color saturation, fruit shape, surface smoothness, etc.).

[0159] In some embodiments of the present invention, generating the maturity assessment result of the food based on the multiple maturity assessment indicators includes: determining weight information of the multiple maturity assessment indicators; and performing weighted summation of the multiple maturity assessment indicators according to the weight information to obtain the maturity assessment result of the food.

[0160] Among them, the weight information of the maturity assessment index can be set with different weights according to the degree of influence on maturity, such as the weight of color saturation is 1, the weight of fruit shape is 2, and the weight of surface smoothness is 3. The weighted sum of these three maturity assessment indexes is performed to obtain the maturity assessment result of the food:

[0161] Food maturity assessment result = x1 color saturation + x2 fruit shape + x3 surface smoothness.

[0162] Step 305: Generate a recommended processing solution for the food according to the maturity evaluation result, and display the maturity evaluation result and the recommended processing solution for the food through the user interaction interface of the refrigerator.

[0163] In actual applications, the user interaction interface can display the food maturity assessment results in real time through a high-definition display screen, and provide recommended food processing solutions (such as eating suggestions); secondly, the user interaction interface is also equipped with historical detection records to better manage the storage and consumption of fruits and vegetables in the refrigerator.

[0164] For example, the apples purchased on October 11 are overripe and their taste may be affected, so please eat them within two days. The cucumbers purchased on October 11 are spoiled and their effects on your health may be affected, so please do not eat them.

[0165] As some examples, the user interface may also display the food maturity assessment results and recommended treatment plans in a table, such as Table 1:

[0166]

[0167] Table 1

[0168] In some examples, the purchase time and the storage time can be determined based on the image data acquired by the camera, and the user can also adjust the purchase time and the storage time by himself through the user interaction interface.

[0169] In an embodiment of the present invention, current image data of food in a refrigerator is acquired; key feature information of the food is determined from the current image data; multiple maturity assessment indicators are determined based on the key feature information; maturity assessment results of the food are generated based on the multiple maturity assessment indicators; recommended processing solutions for the food are generated based on the maturity assessment results, and the maturity assessment results and recommended processing solutions for the food are displayed through a user interaction interface of the refrigerator. This achieves the goal of acquiring current image data of the food in the refrigerator and further processing the data to assess the maturity of the food, thereby avoiding errors in human judgment of food maturity, improving the quality of life of users and reducing food waste.

[0170] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0171] Reference Figure 4 , shows a schematic diagram of the structure of a device for detecting food in a refrigerator provided by some embodiments of the present invention, which may specifically include the following modules:

[0172] The current image data acquisition module 401 is used to acquire the current image data of the food in the refrigerator;

[0173] A key feature information determination module 402, used to determine the key feature information of the food from the current image data;

[0174] A maturity assessment index determination module 403 is used to determine a plurality of maturity assessment indexes according to the key feature information;

[0175] The maturity assessment result generating module 404 is used to generate the maturity assessment result of the food according to the multiple maturity assessment indicators.

[0176] In some embodiments of the present invention, the device further comprises:

[0177] The interactive interface recommendation module is used to generate a recommended processing solution for the food according to the maturity assessment result, and to display the maturity assessment result and the recommended processing solution for the food through the user interactive interface of the refrigerator.

[0178] In some embodiments of the present invention, the maturity assessment result generating module 404 includes:

[0179] A weight information determination submodule, used to determine the weight information of the plurality of maturity assessment indicators;

[0180] The weighted summation submodule is used to perform weighted summation on the multiple maturity assessment indicators according to the weight information to obtain the maturity assessment result of the food.

[0181] In some embodiments of the present invention, the maturity assessment index includes any one of the following: color saturation, fruit shape, and surface smoothness.

[0182] In some embodiments of the present invention, the device further comprises:

[0183] The defogging processing module is used to perform defogging processing on the current image data.

[0184] In some embodiments of the present invention, the device further comprises:

[0185] The normalization processing module is used to perform normalization processing on the current image data.

[0186] In some embodiments of the present invention, the key feature information determination module 402 includes:

[0187] The target detection submodule is used to input the current image data into a pre-trained neural network model, perform target detection on the current image data through the neural network model, and obtain key feature information of the food.

[0188] In some embodiments of the present invention, the key feature information includes any one or more of the following: color information, shape information, and texture information.

[0189] In some embodiments of the present invention, the device further comprises:

[0190] A sample image data acquisition module, used to acquire sample image data, wherein the sample image data includes image data of different lighting scenes;

[0191] The model training module is used to train a preset neural network model using the sample image data.

[0192] In some embodiments of the present invention, the device further comprises:

[0193] The first sample image data processing module is used to perform a defogging process and / or a normalization process on the sample image data.

[0194] The second sample image data processing module is used to perform expansion processing on the sample image data.

[0195] In an embodiment of the present invention, by acquiring current image data of food in a refrigerator; determining key feature information of the food from the current image data; determining multiple maturity assessment indicators based on the key feature information; and generating a maturity assessment result of the food based on the multiple maturity assessment indicators, it is achieved that by acquiring current image data of food in a refrigerator and further processing the data to assess the maturity of the food, errors in human judgment of food maturity are avoided, the quality of life of users can be improved and food waste can be reduced.

[0196] Some embodiments of the present invention further provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the above method is implemented when the computer program is executed by the processor.

[0197] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0198] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above method when executed by a processor.

[0199] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0201] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0202] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0203] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0207] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0208] The above is a detailed introduction to the provided method, device, equipment, medium and product for detecting food in a refrigerator. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technicians in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting food in a refrigerator, characterized in that: The method comprises: Get the current image data of the food in the refrigerator; Determining key feature information of the food from the current image data; Determining a plurality of maturity assessment indicators based on the key characteristic information; A maturity assessment result of the food is generated according to the multiple maturity assessment indicators.

2. The method according to claim 1, characterized in that Also includes: A recommended processing solution for the food is generated according to the maturity assessment result, and the maturity assessment result and the recommended processing solution for the food are displayed through the user interaction interface of the refrigerator.

3. The method according to claim 1 or 2, characterized in that: Generating a maturity evaluation result of the food according to the plurality of maturity evaluation indicators comprises: Determining weight information of the plurality of maturity assessment indicators; According to the weight information, the plurality of maturity assessment indicators are weighted and summed to obtain a maturity assessment result of the food.

4. The method according to claim 3, characterized in that The maturity evaluation index includes any one of the following: color saturation, fruit shape, and surface smoothness.

5. The method according to claim 1 or 2, characterized in that: Before determining the key feature information of the food from the current image data, the method further includes: Performing defogging processing on the current image data.

6. The method according to claim 1 or 2, characterized in that: Before determining the key feature information of the food from the current image data, the method further includes: The current image data is normalized.

7. The method according to claim 1 or 2, characterized in that: Determining the key feature information of the food from the current image data includes: The current image data is input into a pre-trained neural network model, and target detection is performed on the current image data through the neural network model to obtain key feature information of the food.

8. The method according to claim 7, characterized in that The key feature information includes any one or more of the following: color information, shape information, and texture information.

9. The method according to claim 7, characterized in that: Before obtaining the current image data of the food in the refrigerator, the method further includes: Acquire sample image data, wherein the sample image data includes image data of different lighting scenes; The sample image data is used to train a preset neural network model.

10. The method according to claim 9, characterized in that Before the sample image data set is used to train the preset neural network model, the method further includes: Performing defogging and / or normalization processing on the sample image data; Before using the sample image data set to train the preset neural network model, the method further includes: expanding the sample image data.

11. A device for detecting food in a refrigerator, characterized in that: The device comprises: A current image data acquisition module, used to acquire current image data of food in the refrigerator; A key feature information determination module, used to determine the key feature information of the food from the current image data; A maturity assessment indicator determination module, used to determine a plurality of maturity assessment indicators according to the key feature information; The maturity assessment result generating module is used to generate the maturity assessment result of the food according to the multiple maturity assessment indicators.

12. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by the processor.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images