Food and beverage identification method and device for intelligent refrigerator

Abstract

The invention is suitable for the technical field of image recognition, and provides a food and beverage recognition method and device for an intelligent refrigerator, and the method comprises the steps: collecting an original image of a storage platform in the intelligent refrigerator, obtaining preset article information of different image areas in the original image and a target standard article image corresponding to the preset article information; removing the object placing platform area image in the original image to obtain a plurality of object area images; if the target similarity is greater than a first threshold value, determining that the article area image is a preset article; and if the target similarity is not greater than a first threshold value, identifying article information in the article region image according to article contour features in the article region image. According to the method, through multi-step image processing and similarity calculation, consumption of a large amount of calculation resources is avoided, real-time training and reasoning of a deep learning model do not need to be carried out, and therefore the requirement for the calculation resources is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a method and device for identifying food and beverages in a smart refrigerator. Background Art

[0002] A smart vending refrigerator is a device that combines IoT technology, artificial intelligence, and automatic vending functions, and is commonly used in retail, offices, and public places. It can automatically vend a variety of goods, such as beverages, snacks, fast food, medicines, etc.

[0003] With the continuous development of image recognition technology and artificial intelligence, smart vending refrigerators have gradually acquired the ability to automatically identify items stored in the refrigerator through image processing. In the prior art, some smart refrigerators use image recognition technology to obtain images of the inside of the refrigerator through a camera, and combine them with a pre-set item information library to determine the type of items in the refrigerator through image feature matching.

[0004] In order to improve the accuracy of food and beverage recognition in smart refrigerators, there are also attempts in the prior art to enhance the robustness of object recognition through multi-image processing technology, deep learning algorithms, etc. However, these methods often have the disadvantages of slow processing speed and require a large amount of data support and training. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for identifying food and beverages in a smart refrigerator to solve the technical problems of slow processing speed and the need for a large amount of data support and training in image processing technology.

[0006] A first aspect of an embodiment of the present invention provides a method for identifying food and beverages in a smart refrigerator, the method comprising:

[0007] Collecting an original image of a storage platform in a smart refrigerator, and obtaining preset item information of different image areas in the original image and a target standard item image corresponding to the preset item information; the items include beverages and food;

[0008] Eliminating the storage platform area image in the original image to obtain multiple object area images;

[0009] Calculating the target similarity between the object region image and the target standard object image;

[0010] If the target similarity is greater than a first threshold, confirming that the object region image is a preset object;

[0011] If the target similarity is not greater than the first threshold, the object information in the object area image is identified according to the object contour features in the object area image.

[0012] Furthermore, the step of removing the storage platform area image in the original image to obtain multiple object area images includes:

[0013] Acquire pixel information corresponding to the storage platform area, and remove the storage platform area image in the original image according to the pixel information to obtain multiple initial image areas;

[0014] Obtaining the pixel number interval corresponding to the target standard object image;

[0015] If the initial image area is within the pixel number interval, the initial image area is used as the object area image;

[0016] If the initial image area exceeds the pixel number interval, segmentation processing is performed according to the geometric features of the initial image area to obtain the object area image.

[0017] Furthermore, if the initial image area exceeds the pixel number interval, segmenting the initial image area according to geometric features to obtain the object area image comprises:

[0018] If the preset object has a bottle cap structure, extracting a first center position corresponding to the bottle cap structure in the initial image area according to pixel value information corresponding to the bottle cap structure;

[0019] Extracting a first midpoint position between the two first center positions and a first straight line formed by the two first center positions;

[0020] At the first midpoint position, dividing the initial image area into two object area images in a direction perpendicular to the first straight line;

[0021] If the preset object is a can structure, the pixel information corresponding to the can top or can bottom is obtained;

[0022] Extracting second center positions of two tank top regions or tank bottom regions in the initial image region according to pixel information corresponding to the tank top or the tank bottom;

[0023] Extracting a second midpoint position between the two second center positions and a second straight line formed by the two second center positions;

[0024] At the second midpoint position, the initial image area is divided into two object area images in a direction perpendicular to the second straight line.

[0025] Furthermore, the step of calculating the target similarity between the object region image and the target standard object image comprises:

[0026] Obtain pixel information of the sealing structure and pixel information of the main structure corresponding to the preset object information;

[0027] Based on the pixel information of the sealing structure and the pixel information of the main structure, extracting a first sealing structure region and a first main structure region in the object region image; the first sealing structure includes a bottle cap or a can top, and the first main structure includes a bottle body or a can body;

[0028] Extracting a second sealing structure region and a second main structure region in the target standard article image based on the pixel information of the sealing structure and the pixel information of the main structure;

[0029] The target similarity between the object area image and the target standard object image is calculated based on the first sealing structure area size corresponding to the first sealing structure area, the second sealing structure area size corresponding to the second sealing structure area, the first main structure area size corresponding to the first main structure area, the second main structure area size corresponding to the second main structure area, the first area ratio and the second area ratio; the first area ratio includes the ratio between the first sealing structure area size and the first main structure area size, and the second area ratio includes the ratio between the second sealing structure area size and the second main structure area size.

[0030] Further, the step of calculating the target similarity between the object area image and the target standard object image according to the first sealing structure area size corresponding to the first sealing structure area, the second sealing structure area size corresponding to the second sealing structure area, the first main structure area size corresponding to the first main structure area, the second main structure area size corresponding to the second main structure area, the first area ratio and the second area ratio comprises:

[0031] Constructing the first sealing structure area size, the first main structure area size and the first area ratio into a first feature vector;

[0032] The size of the second sealing structure area, the size of the second main structure area and the second area ratio are constructed as a second feature vector;

[0033] The similarity between the first feature vector and the second feature vector is used as the target similarity.

[0034] Furthermore, if the target similarity is not greater than a first threshold, the step of identifying the object information in the object area image according to the object contour features in the object area image includes:

[0035] Extracting the number of edge pixels of the object area image;

[0036] If the number of edge pixels is less than the second threshold, a third feature vector corresponding to other standard object images is obtained; the elements in the third feature vector include the size of the sealing structure area, the size of the main structure area and the area ratio;

[0037] If the similarity between the first feature vector and the third feature vector is greater than a first threshold, then confirming that the object region image is object information corresponding to other standard object images;

[0038] If the number of edge pixels is not less than the second threshold, the object area image is sent to a cloud server, and the object information in the object area image is identified by the cloud server.

[0039] Furthermore, if the number of edge pixels is not less than a second threshold, the object region image is sent to a cloud server, and the step of identifying the object information in the object region image through the cloud server includes:

[0040] If the number of edge pixels is not less than a second threshold, sending the object area image to a cloud server;

[0041] The cloud server extracts texture features in the object area image through a gray level co-occurrence matrix;

[0042] The texture features are input into a pre-trained recognition model to obtain object information output by the recognition model.

[0043] A second aspect of an embodiment of the present invention provides a food and beverage identification device for a smart refrigerator, comprising:

[0044] A collection unit, used to collect the original image of the storage platform in the smart refrigerator, and obtain the preset object information of different image areas in the original image and the target standard object image corresponding to the preset object information; the objects include beverages and food;

[0045] A removal unit, used to remove the storage platform area image in the original image to obtain multiple object area images;

[0046] A calculation unit, used for calculating the target similarity between the object region image and the target standard object image;

[0047] A first judgment unit, configured to confirm that the object region image is a preset object if the target similarity is greater than a first threshold;

[0048] The second judgment unit is configured to identify the object information in the object area image according to the object contour features in the object area image if the target similarity is not greater than the first threshold.

[0049] The third aspect of an embodiment of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the food and beverage identification method for the smart refrigerator described in the first aspect are implemented.

[0050] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the food and beverage identification method for the smart refrigerator described in the first aspect.

[0051] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: by collecting the original image of the storage platform of the smart refrigerator and quickly identifying the object according to the preset object information in the image area, deep training of a large amount of image data is avoided, thereby significantly reducing the processing time. In particular, the storage platform area is eliminated in the original image, which can effectively reduce the complexity of image processing, thereby accelerating the object recognition process. The target similarity algorithm is used to calculate the similarity between the object area image and the standard target object image to determine whether the object meets the preset standard. When the similarity exceeds the set first threshold, the object category can be quickly confirmed to ensure the accuracy of recognition; when the similarity does not meet the standard, the present invention adopts a recognition method based on the contour feature of the object, and further improves the reliability of object recognition through morphological analysis of the object area, which is particularly suitable for foods and beverages with special shapes or appearances. Traditional image processing methods rely on a large amount of labeled data for deep learning training, and usually require a lot of time and computing resources. However, the present invention can efficiently complete the recognition task under a smaller data set by matching the preset object information with the standard object image, reducing the dependence on large-scale data sets and lowering the threshold for technical application. Since the present invention combines multiple technical means of image area elimination, target similarity comparison and object contour feature recognition, it can effectively deal with the impact of changes in the position, angle, light and other aspects of the objects in the refrigerator on image recognition, and has strong adaptability. Whether the objects are placed vertically or tilted, they can be accurately identified. The method of the present invention avoids a large amount of computing resource consumption through multi-step image processing and similarity calculation, and does not require real-time training and reasoning of deep learning models, thereby reducing the demand for computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A schematic flow chart of a method for identifying food and beverages in a smart refrigerator provided by the present invention is shown;

[0054] Figure 2 A schematic diagram of a food and beverage identification device for a smart refrigerator provided by an embodiment of the present invention is shown;

[0055] Figure 3 A schematic diagram of a terminal device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0056] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0057] The embodiments of the present invention provide a method and device for identifying food and beverages in a smart refrigerator to solve the technical problems of slow processing speed and the need for a large amount of data support and training in image processing technology.

[0058] First, the present invention provides a method for identifying food and beverages in a smart refrigerator. Figure 1 , Figure 1 FIG. 1 is a schematic flow chart of a method for identifying food and beverages in a smart refrigerator provided by the present invention. Figure 1 As shown, the food and beverage identification method of the smart refrigerator may include the following steps:

[0059] Step 101: collecting an original image of a storage platform in a smart refrigerator, and obtaining preset item information of different image areas in the original image and a target standard item image corresponding to the preset item information; the items include beverages and food;

[0060] The smart refrigerator has multiple storage platforms, each of which has multiple storage areas. Different storage areas are used to store different beverages or foods, and the same storage area is used to store the same beverage or food. The bottom color information of different storage platforms or storage areas is different (to avoid conflict with the color information of the surface of the beverage or food).

[0061] Each containing area corresponds to a fixed image area in the original image. The user sets preset object information for different image areas in advance.

[0062] Since smart refrigerators are in actual vending scenarios and involve replenishment and purchasing operations, and there may also be sales anomalies due to mistakes made by the replenisher or the buyer inadvertently placing the wrong beverage or food in the wrong storage area, it is necessary to periodically identify the beverages or food in the storage area and match their corresponding item information in order to correctly track and sell them later.

[0063] The smart refrigerator uses a camera to collect the original image of the refrigerator's built-in platform. From the collected original image, the system will mark different image areas and correspond these areas to preset item information (such as "mineral water", "XX Forest" and "milk", etc.). Each item information will be associated with a target standard item image, which usually refers to the typical style image of the item as a reference standard for subsequent recognition.

[0064] For example, for the item "bottled water", the preset item information is "bottled water", and the target standard item image is a standard bottled water image. The system confirms the item by comparing the similarity between the actual image area and the target standard image.

[0065] Step 102: Eliminate the storage platform area image in the original image to obtain multiple object area images;

[0066] Since the bottom color information of different storage platforms or storage areas is different, the storage platform area image in the original image can be eliminated through the bottom color information of the storage platform or storage area to obtain multiple image areas. In actual scenes, if the positions of multiple beverages or foods are too close, there may be multiple beverages or foods merged in the same image area, so it is also necessary to segment different image areas. The specific logic is as follows:

[0067] Specifically, step 102 specifically includes steps 1021 to 1024:

[0068] Step 1021: Obtain pixel information corresponding to the storage platform area, and remove the storage platform area image in the original image according to the pixel information to obtain multiple initial image areas;

[0069] It is worth noting that when placing different beverages on the placement platform, it is necessary to avoid using the same or similar color as the bottom color of the placement platform and the surface color of the beverage to avoid affecting the recognition accuracy.

[0070] Step 1022: Obtain the pixel number interval corresponding to the target standard object image;

[0071] The pixel number interval is used to screen the situation where multiple beverages or foods are integrated into the same image area. It is understandable that the number of pixels corresponding to a single beverage is within the pixel number interval, while the integration of multiple beverages or foods will far exceed the pixel number interval. The pixel number interval can be determined based on the number of pixels occupied by different beverages in the actual application environment.

[0072] Step 1023: If the initial image area is within the pixel number interval, the initial image area is used as the object area image;

[0073] If the number of pixels in the initial image area falls within the pixel number range of the target standard object image, it means that the image in this area meets the size range of the standard object, and the area is considered to be an object area image.

[0074] Step 1024: If the initial image area exceeds the pixel number range, segmentation processing is performed according to the geometric features of the initial image area to obtain the object area image.

[0075] If the number of pixels in a certain initial image area exceeds the pixel number range of the target standard object, it means that the area may contain multiple objects or parts of an object. It is necessary to extract multiple object areas through geometric feature segmentation based on the image area to ensure that an accurate single object area image is finally obtained. The specific segmentation principle is as follows:

[0076] Specifically, step 1024 specifically includes steps A1 to A7:

[0077] Step A1: if the preset object has a bottle cap structure, extracting a first center position corresponding to the bottle cap structure in the initial image area according to pixel value information corresponding to the bottle cap structure;

[0078] Since the color information of the bottle cap structure and the bottle body is often quite different, the image area corresponding to the pixel value information can be extracted based on the pixel value information corresponding to the bottle cap structure to obtain the image area corresponding to the bottle cap structure and extract the first center position corresponding to the bottle cap structure.

[0079] Step A2: extracting a first midpoint position between two of the first center positions and a first straight line formed by the two first center positions;

[0080] After extracting the two center points of the bottle cap, the midpoint between the two points (ie, the first midpoint) is calculated, and the straight line formed by the two center points (ie, the first straight line) is determined.

[0081] Step A3: At the first midpoint position, dividing the initial image area into two object area images in a direction perpendicular to the first straight line;

[0082] This step is mainly used to segment the two bottle regions to ensure that each bottle is identified separately.

[0083] Step A4: if the preset object is a can structure, then obtain pixel information corresponding to the can top or can bottom;

[0084] Step A5: extracting the second center positions of the two can top areas or can bottom areas in the initial image area according to the pixel information corresponding to the can top or the can bottom;

[0085] The structure of a can includes a can top, a can bottom and a can body. There is often an obvious color difference between the can top and the can bottom and the can body, so based on the pixel value information corresponding to the can top or the can bottom, the image area corresponding to the can top or the can bottom can be extracted, and the second center position corresponding to the can top or the can bottom can be extracted.

[0086] Step A6: extracting a second midpoint position between the two second center positions and a second straight line formed by the two second center positions;

[0087] Step A7: At the second midpoint position, the initial image area is divided into two object area images in a direction perpendicular to the second straight line.

[0088] This step is mainly used to segment the area of ​​the two cans to ensure that each can is identified separately.

[0089] The above steps determine the first center position of the bottle cap structure by extracting the pixel value information of the bottle cap structure. Then, the first midpoint and the first straight line between the two bottle cap center positions are calculated, and the segmentation is performed at the first midpoint position in a direction perpendicular to the first straight line. This segmentation method can accurately segment the object area image into two parts, extract the bottle cap area and the bottle body area, and thus optimize the recognition effect of the object area. For the can structure, by obtaining the pixel information of the can top or the can bottom, the second center position of the two can top areas or the can bottom areas is extracted. Similarly, the second midpoint and the second straight line between the two second center positions are calculated, and at the second midpoint position, the initial image area is segmented into two object area images in a direction perpendicular to the second straight line. This process effectively identifies and segments the top and bottom areas of the can, ensuring the accurate extraction of the object area in the image. Segmentation based on the geometric features of the bottle cap and the can effectively avoids errors in the object recognition process, especially when the object shape is complex or partially overlapped, and can accurately distinguish different object areas. Whether it is the bottle cap structure of a bottled beverage or the top and bottom areas of a can, the method of the present invention can flexibly segment through different geometric feature extractions and has strong adaptability. By directional segmentation of the object structure, the clear division of the object area in the refrigerator is ensured, reducing the risk of misidentification and missed identification.

[0090] In this embodiment, the method accurately locates the storage platform area inside the refrigerator by acquiring pixel information corresponding to the storage platform area, and removes the area from the original image to obtain multiple initial image areas. This process effectively eliminates the interfering background area in the refrigerator image, making the subsequent extraction of the object area more accurate. Then, the initial image area is screened using the pixel number interval of the target standard object image. For areas that meet the pixel number interval, they are directly identified as object area images. This process is based on the standard of the number of pixels, and can effectively filter out areas with abnormal sizes or non-target objects, reducing the risk of misidentification. For initial image areas that do not meet the pixel number interval, segmentation processing is further performed by analyzing their geometric features to accurately extract the object area image. This step can effectively identify objects with complex shapes or partial occlusions, avoiding situations where simple pixel statistics methods cannot cope with.

[0091] Step 103: Calculating the target similarity between the object region image and the target standard object image;

[0092] Specifically, step 103 specifically includes steps 1031 to 1034:

[0093] Step 1031: Obtain pixel information of the sealing structure and pixel information of the main body structure corresponding to the preset object information;

[0094] Sealing structures are the sealing parts of an item, such as a cap (on a bottle) or a top (on a can). These structures are usually the top of the item and have a specific shape and size.

[0095] The main structure refers to the main part of the object, such as the body of a bottle or can. These areas are usually longer and more regular in shape than the sealing structure.

[0096] Step 1032: extracting a first sealing structure region and a first main structure region in the object region image based on the pixel information of the sealing structure and the pixel information of the main structure; the first sealing structure includes a bottle cap or a can top, and the first main structure includes a bottle body or a can body;

[0097] The area where the pixel information of the sealing structure is located in the object area image is extracted to obtain the first sealing structure area. The area where the pixel information of the main structure is located in the object area image is extracted to obtain the first main structure area.

[0098] Step 1033: extracting a second sealing structure region and a second main body structure region in the target standard article image based on the pixel information of the sealing structure and the pixel information of the main body structure;

[0099] The area where the pixel information of the sealing structure is located in the object area image is extracted to obtain the second sealing structure area. The area where the pixel information of the main structure is located in the object area image is extracted to obtain the second main structure area.

[0100] Step 1034: Calculate the target similarity between the object area image and the target standard object image based on the first sealing structure area size corresponding to the first sealing structure area, the second sealing structure area size corresponding to the second sealing structure area, the first main structure area size corresponding to the first main structure area, the second main structure area size corresponding to the second main structure area, the first area ratio and the second area ratio; the first area ratio includes the ratio between the first sealing structure area size and the first main structure area size, and the second area ratio includes the ratio between the second sealing structure area size and the second main structure area size.

[0101] Specifically, step 1034 includes steps B1 to B3:

[0102] Step B1: constructing the first sealing structure area size, the first main structure area size and the first area ratio into a first feature vector;

[0103] Step B2: constructing the second sealing structure area size, the second main structure area size and the second area ratio into a second feature vector;

[0104] Step B3: taking the similarity between the first feature vector and the second feature vector as the target similarity.

[0105] The above steps extract key structural areas in the object area image, such as the first sealing structure area (such as the bottle cap or can top), the first main structure area (such as the bottle body or can body), and construct the first feature vector based on the size of these areas and the first area ratio (the ratio of the size of the first sealing structure area to the size of the first main structure area). At the same time, the corresponding areas in the target standard object image, such as the second sealing structure area and the second main structure area, are extracted, and the second feature vector is constructed based on the size of these areas and the second area ratio (the ratio of the size of the second sealing structure area to the size of the second main structure area). In this way, the present invention converts the key structural features of the object into a high-dimensional vector for subsequent calculation. Through the constructed first feature vector and second feature vector, the present invention further calculates the similarity between the two. The similarity measures the structural similarity between the object area image and the target standard object image, thereby realizing accurate identification and classification of the object. The basis of similarity calculation is the comprehensive consideration of area size and ratio, which ensures the comprehensive matching of the morphological features of the object. The structural features are processed by vectorization, so that the object identification not only depends on a single area information, but combines the features of multiple dimensions (such as structure size and ratio). This integration of multi-dimensional features can significantly improve the accuracy of similarity calculations and ensure effective distinction and identification between different objects.

[0106] In this embodiment, by obtaining the pixel information of the sealing structure (such as a bottle cap or a can top) and the main structure (such as a bottle body or a can body) of the object, the first sealing structure area and the first main structure area in the object area image are accurately extracted. At the same time, according to the sealing structure and main structure information of the target standard object image, the corresponding second sealing structure area and the second main structure area are extracted. This process ensures the effective capture of the object structure and provides an accurate basis for the subsequent similarity calculation. According to the extracted area size information, the first area ratio (the ratio of the first sealing structure area to the first main structure area size) and the second area ratio (the ratio of the second sealing structure area to the second main structure area size) are calculated. By comprehensively considering the relationship between these ratios and the area size, the present invention can accurately measure the target similarity between the object area image and the target standard object image. This method not only takes into account the overall size of the object, but also particularly emphasizes the proportional relationship between the various structural parts, further improving the accuracy of the similarity calculation. By combining the above-mentioned area ratios, the target similarity between the object area image and the target standard object image is calculated. This calculation process effectively compares the structural features of the object with the corresponding relationship of the standard object, and can accurately determine the type of object, thereby supporting food and beverage recognition in smart refrigerators.

[0107] As an optional embodiment of the present application, since some beverages have similar shapes and color features, if only vector similarity calculation is used, there may be a technical problem of low recognition accuracy. In order to improve the recognition calculation, the present application also provides another method for calculating target similarity, the specific logic is as follows:

[0108] Substitute the first sealing structure area size, the first main structure area size, the first area ratio, the second sealing structure area size, the second main structure area size, the first area ratio and the second area ratio into the following mathematical model to obtain the target similarity:

[0109]

[0110] Among them, S represents the target similarity, A1 represents the size of the first sealing structure area, A2 represents the size of the second sealing structure area, B1 represents the size of the first main structure area, B2 represents the size of the second main structure area, R1 represents the first area ratio, R2 represents the second area ratio, w1 represents the first weight factor, and W2 represents the second weight factor.

[0111] In many cases, the size similarity of structural regions in different object images can reflect the similarity of object morphology. The sealing structure and main structure of an object have a certain proportional relationship visually, and the similarity between them can be evaluated by comparing the size difference of the corresponding regions in the two images.

[0112] By calculating the relative size difference between the sealed structure and the main structure area, it is possible to evaluate whether the structures in the image are similar. If the size difference between the corresponding areas in the current object and the standard object image is small, then they are relatively similar in shape.

[0113] A1 and A2 are the sizes of the sealing structure areas of the two objects, and B1 and B2 are the sizes of the main structure areas. Relative differences are used in calculations, that is, the maximum value is normalized to avoid the deviation of the absolute value on the similarity. This method allows the difference to be quantified within the range of 0 to 1 regardless of the actual size of the area. The closer the similarity is to 1, the more similar the two areas are, and vice versa.

[0114] The ratio is used to measure the relative relationship between two key structural areas in the object image. The ratio of the sealing structure to the main structure is fixed, and the ratio in the current object image should be as close to this standard value as possible. If the ratio of the two is very different, it means that the structural relationship between the current object and the standard object is quite different.

[0115] By comparing the ratio of the seal structure to the main structure of the current object and the standard object, we can further measure the similarity of their structural forms. The difference in the ratio reflects the degree of morphological difference between the two.

[0116] In order to comprehensively consider the differences in region size and ratio, we introduced weighted terms to represent the importance of region size similarity and ratio similarity in the final calculation.

[0117] In practical applications, the size difference and ratio difference of regions may have different effects on similarity, so we set different weights for these two factors. By taking the weighted sum approach, we can ensure that the contribution of these two factors to the final similarity meets the requirements.

[0118] In order to ensure that the final similarity value is within a standard range (between 0 and 1), the comprehensive similarity value needs to be normalized. This is because after weighted summation, the similarity obtained may exceed the range of 0 to 1, so it needs to be normalized to make the calculation result easier to interpret.

[0119] By normalizing, we standardize the calculated similarities so that even when the region sizes and ratios between different items vary greatly, the results can still be compared on a uniform scale.

[0120] Step 104: If the target similarity is greater than a first threshold, confirming that the object region image is a preset object;

[0121] If the target similarity is greater than the first threshold, it means that the similarity between the object area image and the target standard object image is high enough, and the system confirms that the object area image belongs to a preset object (e.g., bottled water, juice, etc.). At this point, the system can directly identify the object and perform further processing (such as updating the inventory in the refrigerator or reminding the user).

[0122] Step 105: If the target similarity is not greater than the first threshold, identifying the object information in the object region image according to the object contour features in the object region image.

[0123] If the target similarity is not greater than the first threshold, it means that the similarity between the object area and the standard object image is low. In this case, the system uses the contour features in the object area image to identify the object. The specific logic is as follows:

[0124] Specifically, step 105 specifically includes steps 1051 to 1054:

[0125] Step 1051: If the target similarity is not greater than a first threshold, extracting the number of edge pixels of the object area image;

[0126] Step 1052: if the number of edge pixels is less than the second threshold, then obtaining a third feature vector corresponding to other standard object images; the elements in the third feature vector include the size of the sealing structure area, the size of the main structure area and the area ratio;

[0127] During the beverage sales process, the beverage may be placed horizontally or tilted. When the beverage is placed horizontally or tilted, its outline is longer. When the beverage is in a normal upright state, its outline is shorter. Therefore, when the number of edge pixels is less than the second threshold, other standard object images can be obtained to determine whether other beverages appear in the current accommodation area (for example, cola appears in the mineral water area, etc.). The elements in the third eigenvector include but are not limited to the size of the sealing structure area, the size of the main structure area, and the area ratio.

[0128] Step 1053: If the similarity between the first feature vector and the third feature vector is greater than the first threshold, confirming that the object region image is object information corresponding to other standard object images;

[0129] The system calculates the similarity between the first feature vector (based on the sealing structure, main structure and area ratio) of the object area image and the third feature vector of these other standard object images. If the similarity between the first feature vector and the third feature vector is greater than the first threshold, it means that the features of the object area image are similar to those of the other standard object images, and the system can confirm that the object area image belongs to the object information corresponding to these other standard object images.

[0130] The calculation method of the third eigenvector is the same as that of the first eigenvector, which will not be described in detail here.

[0131] By comprehensively considering the target similarity, the number of edge pixels, and the similarity of feature vectors, this method can carefully distinguish the different features of objects during the recognition process, thereby improving the accuracy of recognition. When the target similarity is low, the dual strategy of edge pixel number analysis and standard object feature vector comparison can ensure that even in the case of low similarity, object recognition can still be completed through reasonable judgment, reducing the risk of misidentification. By setting reasonable thresholds (first threshold, second threshold), this method can flexibly adjust the judgment conditions in the recognition process, optimize the processing flow, avoid redundant calculations in complex situations, and thus improve recognition efficiency.

[0132] Step 1054: If the number of edge pixels is not less than the second threshold, the object region image is sent to a cloud server, and the object information in the object region image is identified by the cloud server.

[0133] If the number of edge pixels is not less than the second threshold, it means that the beverage may be placed horizontally or tilted. This situation requires calculations with higher recognition accuracy, which is not supported by local computing power. Cloud servers usually have more powerful computing resources and storage capabilities, and can perform more complex image processing and analysis tasks. Therefore, the object area image can be sent to the cloud server, and the cloud server can identify the object information in the object area image. The recognition logic of the cloud server is as follows:

[0134] Specifically, step 1054 includes steps C1 to C3:

[0135] Step C1: if the number of edge pixels is not less than a second threshold, sending the object area image to a cloud server;

[0136] Step C2: the cloud server extracts texture features in the object area image through a gray level co-occurrence matrix;

[0137] The cloud server uses the gray-level co-occurrence matrix to extract texture features in the item region image. The gray-level co-occurrence matrix is ​​a method to describe the spatial relationship between the gray levels of an image. It can capture texture information in the image, such as roughness, smoothness, directionality, etc.

[0138] Specifically, the gray level co-occurrence matrix calculates the co-occurrence frequency of pixel gray value pairs in an image within a certain spatial distance. By analyzing these co-occurrence frequencies, a variety of texture features can be extracted, such as:

[0139] Contrast: Indicates the degree of local change in texture.

[0140] Correlation: Describes the correlation of gray levels in a texture.

[0141] Energy: Indicates the uniformity of image texture.

[0142] Homogeneity: Indicates whether the texture of the image is smooth and consistent.

[0143] Step C3: input the texture feature into a pre-trained recognition model to obtain the object information output by the recognition model.

[0144] The cloud server inputs the texture features extracted from the object area image into a pre-trained recognition model for analysis. This recognition model is usually trained based on machine learning or deep learning algorithms (such as convolutional neural network CNN, support vector machine SVM, etc.).

[0145] Training process: During the training phase, the recognition model uses a large amount of labeled standard object image data to learn and understand the relationship between the texture features and other attributes of different objects. Through training, the model can determine the type of object from the input texture features.

[0146] Recognition process: In actual applications, after the cloud server receives the input texture features, the model uses prediction and classification algorithms to obtain the object category information corresponding to the object area image.

[0147] Item information output: Ultimately, the pre-trained recognition model outputs the item recognition result, that is, the item information, including the type of item (such as beverages, food, etc.) and detailed information such as the specific name or model. This information can be fed back to the smart refrigerator for further operations, such as updating inventory and setting reminders.

[0148] By extracting texture features through the gray-level co-occurrence matrix, very detailed texture information in the image can be captured, which is especially effective for complex or detailed items (such as packaging with patterns, printed labels, etc.). This method combines the advantages of edge detection and cloud-based deep learning models, and can select the appropriate processing path according to the characteristics of the image, improving the flexibility and accuracy of recognition. Through cloud servers, more powerful computing power and more advanced recognition models can be used to process more complex images, further improving recognition results.

[0149] In this embodiment, by collecting the original image of the storage platform of the smart refrigerator and quickly identifying the preset item information according to the image area, deep training of a large amount of image data is avoided, thereby significantly reducing the processing time. In particular, the storage platform area is eliminated in the original image, which can effectively reduce the complexity of image processing, thereby accelerating the object recognition process. The target similarity algorithm is used to calculate the similarity between the image of the object area and the standard target object image to determine whether the object meets the preset standard. When the similarity exceeds the set first threshold, the object category can be quickly confirmed to ensure the accuracy of recognition; when the similarity does not meet the standard, the present invention adopts a recognition method based on the contour features of the object, and further improves the reliability of object recognition through morphological analysis of the object area, which is particularly suitable for those foods and beverages with special shapes or appearances. Traditional image processing methods rely on a large amount of labeled data for deep learning training, and usually require a lot of time and computing resources. The present invention can efficiently complete the recognition task under a smaller data set by matching the preset object information with the standard object image, reducing the dependence on large-scale data sets and lowering the threshold for technical application. Since the present invention combines multiple technical means of image area elimination, target similarity comparison and object contour feature recognition, it can effectively deal with the impact of changes in the position, angle, light and other aspects of the objects in the refrigerator on image recognition, and has strong adaptability. Whether the objects are placed vertically or tilted, they can be accurately identified. The method of the present invention avoids a large amount of computing resource consumption through multi-step image processing and similarity calculation, and does not require real-time training and reasoning of deep learning models, thereby reducing the demand for computing resources.

[0150] like Figure 2 The present invention provides a food and beverage identification device for a smart refrigerator, see Figure 2 , Figure 2 A schematic diagram of a food and beverage identification device for a smart refrigerator provided by the present invention is shown. Figure 2 The food and beverage identification device of a smart refrigerator includes:

[0151] The acquisition unit 21 is used to acquire the original image of the storage platform in the smart refrigerator, and obtain the preset item information of different image areas in the original image and the target standard item image corresponding to the preset item information; the items include beverages and food;

[0152] A removal unit 22, used to remove the storage platform area image in the original image to obtain multiple object area images;

[0153] A calculation unit 23, used for calculating the target similarity between the object region image and the target standard object image;

[0154] A first judgment unit 24, configured to confirm that the object region image is a preset object if the target similarity is greater than a first threshold;

[0155] The second judgment unit 25 is configured to identify the object information in the object region image according to the object contour features in the object region image if the target similarity is not greater than the first threshold.

[0156] The invention provides a food and beverage identification device for a smart refrigerator. By collecting the original image of the storage platform of the smart refrigerator and presetting the item information according to the image area for rapid identification, the in-depth training of a large amount of image data is avoided, thereby significantly reducing the processing time. In particular, the storage platform area is eliminated in the original image, which can effectively reduce the complexity of image processing, thereby accelerating the object identification process. The target similarity algorithm is adopted to calculate the similarity between the image of the object area and the standard target object image to determine whether the object meets the preset standard. When the similarity exceeds the set first threshold, the object category can be quickly confirmed to ensure the accuracy of identification; when the similarity does not meet the standard, the invention adopts an identification method based on the contour feature of the object, and further improves the reliability of object identification through morphological analysis of the object area, which is particularly suitable for those foods and beverages with special shapes or appearances. Traditional image processing methods rely on a large amount of labeled data for deep learning training, and usually require a lot of time and computing resources. The invention can efficiently complete the identification task under a smaller data set by matching the preset object information with the standard object image, reducing the dependence on large-scale data sets and lowering the threshold for technical application. Since the present invention combines multiple technical means of image area elimination, target similarity comparison and object contour feature recognition, it can effectively deal with the impact of changes in the position, angle, light and other aspects of the objects in the refrigerator on image recognition, and has strong adaptability. Whether the objects are placed vertically or tilted, they can be accurately identified. The method of the present invention avoids a large amount of computing resource consumption through multi-step image processing and similarity calculation, and does not require real-time training and reasoning of deep learning models, thereby reducing the demand for computing resources.

[0157] Figure 3 Schematic diagram of a terminal device provided by an embodiment of the present invention. Figure 3 As shown, a terminal device 3 of this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a food and beverage recognition program for a smart refrigerator. When the processor 30 executes the computer program 32, the steps of each of the above-mentioned food and beverage recognition method embodiments of a smart refrigerator are implemented, such as Figure 1Alternatively, when the processor 30 executes the computer program 32, the functions of each unit in the above-mentioned device embodiments are realized, for example, Figure 2 Function of the unit shown.

[0158] Exemplarily, the computer program 32 may be divided into one or more units, which are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 32 in the terminal device 3. For example, the computer program 32 may be divided into the following specific functions of each unit:

[0159] A collection unit, used to collect the original image of the storage platform in the smart refrigerator, and obtain the preset object information of different image areas in the original image and the target standard object image corresponding to the preset object information; the objects include beverages and food;

[0160] A removal unit, used to remove the storage platform area image in the original image to obtain multiple object area images;

[0161] A calculation unit, used to calculate the target similarity between the object region image and the target standard object image;

[0162] A first judgment unit, configured to confirm that the object region image is a preset object if the target similarity is greater than a first threshold;

[0163] The second judgment unit is configured to identify the object information in the object area image according to the object contour features in the object area image if the target similarity is not greater than the first threshold.

[0164] The terminal device includes but is not limited to a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of a terminal device 3 and does not constitute a limitation on the terminal device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0165] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0166] The memory 31 may be an internal storage unit of the terminal device 3, such as a hard disk or memory of the terminal device 3. The memory 31 may also be an external storage device of the terminal device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the terminal device 3. The memory 31 is used to store the computer program and other programs and data required by the roaming control device. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0167] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0168] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0169] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0170] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0171] An embodiment of the present invention provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0175] In the embodiments provided by the present invention, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed over multiple network units.

[0177] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0178] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0179] As used in the present specification and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to monitoring, depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is monitored" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is monitored" or "in response to monitoring [described condition or event]", depending on the context.

[0180] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0181] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0182] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for identifying food and beverages in a smart refrigerator, characterized in that: The food and beverage identification method of the smart refrigerator includes: Collecting an original image of a storage platform in a smart refrigerator, and obtaining preset item information of different image areas in the original image and a target standard item image corresponding to the preset item information; the items include beverages and food; Eliminating the storage platform area image in the original image to obtain multiple object area images; Calculating the target similarity between the object region image and the target standard object image; If the target similarity is greater than a first threshold, confirming that the object region image is a preset object; If the target similarity is not greater than the first threshold, the object information in the object area image is identified according to the object contour features in the object area image.

2. The method for identifying food and beverages in a smart refrigerator according to claim 1, characterized in that: The step of removing the storage platform area image in the original image to obtain multiple object area images includes: Acquire pixel information corresponding to the storage platform area, and remove the storage platform area image in the original image according to the pixel information to obtain multiple initial image areas; Obtaining the pixel number interval corresponding to the target standard object image; If the initial image area is within the pixel number interval, the initial image area is used as the object area image; If the initial image area exceeds the pixel number interval, segmentation processing is performed according to the geometric features of the initial image area to obtain the object area image.

3. The method for identifying food and beverages in a smart refrigerator according to claim 2, wherein: If the initial image area exceeds the pixel number interval, segmenting the initial image area according to the geometric features of the initial image area to obtain the object area image comprises: If the preset object has a bottle cap structure, extracting a first center position corresponding to the bottle cap structure in the initial image area according to pixel value information corresponding to the bottle cap structure; Extracting a first midpoint position between the two first center positions and a first straight line formed by the two first center positions; At the first midpoint position, dividing the initial image area into two object area images in a direction perpendicular to the first straight line; If the preset object is a can structure, the pixel information corresponding to the can top or can bottom is obtained; Extracting second center positions of two tank top regions or tank bottom regions in the initial image region according to pixel information corresponding to the tank top or the tank bottom; Extracting a second midpoint position between the two second center positions and a second straight line formed by the two second center positions; At the second midpoint position, the initial image area is divided into two object area images in a direction perpendicular to the second straight line.

4. The method for identifying food and beverages in a smart refrigerator according to claim 1, wherein: The step of calculating the target similarity between the object region image and the target standard object image comprises: Obtain pixel information of the sealing structure and pixel information of the main structure corresponding to the preset object information; Based on the pixel information of the sealing structure and the pixel information of the main structure, extracting a first sealing structure region and a first main structure region in the object region image; the first sealing structure includes a bottle cap or a can top, and the first main structure includes a bottle body or a can body; Extracting a second sealing structure region and a second main structure region in the target standard article image based on the pixel information of the sealing structure and the pixel information of the main structure; The target similarity between the object area image and the target standard object image is calculated based on the first sealing structure area size corresponding to the first sealing structure area, the second sealing structure area size corresponding to the second sealing structure area, the first main structure area size corresponding to the first main structure area, the second main structure area size corresponding to the second main structure area, the first area ratio and the second area ratio; the first area ratio includes the ratio between the first sealing structure area size and the first main structure area size, and the second area ratio includes the ratio between the second sealing structure area size and the second main structure area size.

5. The method for identifying food and beverages in a smart refrigerator according to claim 4, characterized in that: The step of calculating the target similarity between the object area image and the target standard object image according to the size of the first sealing structure area corresponding to the first sealing structure area, the size of the second sealing structure area corresponding to the second sealing structure area, the size of the first main body structure area corresponding to the first main body structure area, the size of the second main body structure area corresponding to the second main body structure area, the first area ratio and the second area ratio comprises: Constructing the first sealing structure area size, the first main structure area size and the first area ratio into a first feature vector; The size of the second sealing structure area, the size of the second main structure area and the second area ratio are constructed as a second feature vector; The similarity between the first feature vector and the second feature vector is used as the target similarity.

6. The method for identifying food and beverages in a smart refrigerator according to claim 1, wherein: If the target similarity is not greater than the first threshold, the step of identifying the object information in the object area image according to the object contour features in the object area image includes: If the target similarity is not greater than a first threshold, extracting the number of edge pixels of the object area image; If the number of edge pixels is less than the second threshold, a third feature vector corresponding to other standard object images is obtained; the elements in the third feature vector include the size of the sealing structure area, the size of the main structure area and the area ratio; If the similarity between the first feature vector and the third feature vector is greater than a first threshold, then confirming that the object region image is object information corresponding to other standard object images; If the number of edge pixels is not less than the second threshold, the object area image is sent to a cloud server, and the object information in the object area image is identified by the cloud server.

7. The method for identifying food and beverages in a smart refrigerator according to claim 6, characterized in that: If the number of edge pixels is not less than a second threshold, the object region image is sent to a cloud server, and the step of identifying the object information in the object region image by the cloud server includes: If the number of edge pixels is not less than a second threshold, sending the object area image to a cloud server; The cloud server extracts texture features in the object area image through a gray level co-occurrence matrix; The texture features are input into a pre-trained recognition model to obtain object information output by the recognition model.

8. A food and beverage identification device for a smart refrigerator, characterized in that: The food and beverage identification device of the smart refrigerator comprises: A collection unit, used to collect the original image of the storage platform in the smart refrigerator, and obtain the preset object information of different image areas in the original image and the target standard object image corresponding to the preset object information; the objects include beverages and food; A removal unit, used to remove the storage platform area image in the original image to obtain multiple object area images; A calculation unit, used for calculating the target similarity between the object region image and the target standard object image; A first judgment unit, configured to confirm that the object region image is a preset object if the target similarity is greater than a first threshold; The second judgment unit is configured to identify the object information in the object area image according to the object contour features in the object area image if the target similarity is not greater than the first threshold.

9. A terminal device, characterized in that: The terminal device includes: a memory, a processor, and a food and beverage recognition program for the smart refrigerator stored in the memory and executable on the processor, wherein the food and beverage recognition program for the smart refrigerator is configured to implement the steps in the food and beverage recognition method for the smart refrigerator as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps in the food and beverage identification method of the smart refrigerator as described in any one of claims 1 to 7 are implemented.

Images