A food material recognition method of a smart refrigerator and a related device

Abstract

The application discloses a food material recognition method of an intelligent refrigerator and a related device. The method obtains a to-be-processed image by performing a defogging process on a target image. The stored food materials in the to-be-processed image are recognized. If the food material names of the stored food materials are not recognized, the to-be-processed image is compared with a stored food material image set in terms of similarity. In the above process, the to-be-processed image after the defogging process is recognized first to improve the recognition ability of the stored food materials. Since the stored food material image set is a collection of stored food material images corresponding to each food material stored in the intelligent refrigerator, when the stored food materials after the defogging process cannot be recognized, the to-be-processed image can be compared with the stored food material image set in terms of similarity to obtain a target food material image most similar to the to-be-processed image. Then, the food material name of the stored food material in the to-be-processed image is determined according to the food material corresponding to the target food material image, so that the food material misrecognition rate is reduced.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and device for identifying food items in a smart refrigerator. Background Technology

[0002] With the development of technology, smart refrigerators have gradually entered the homes of many users due to their convenient and diverse intelligent functions. Currently, food identification and entry require visual recognition through the smart refrigerator's built-in camera. This type of technology does not require active user intervention and is more intelligent than refrigerators that require manual or voice input.

[0003] In related technologies, smart refrigerators often use cameras to capture at least one frame of an image of the food during the user's storage and retrieval process. The image is then used to identify the type of food being stored. However, after food has been stored in a smart refrigerator for a period of time, the low temperature can cause it to fog up, frost up, or deform due to freezing. These conditions can prevent the smart refrigerator from correctly identifying the food, resulting in a high rate of misidentification. Summary of the Invention

[0004] This application provides a method and related apparatus for identifying food ingredients in a smart refrigerator. By comparing the similarity between the image to be processed and a set of stored food ingredient images, the target food ingredient image is determined. Then, the name of the food ingredient stored in the image to be processed is determined based on the target food ingredient image, thereby reducing the misidentification rate of food ingredients.

[0005] In a first aspect, embodiments of this application provide a method for identifying food items in a smart refrigerator, the method comprising:

[0006] Acquire a target image, the target image containing stored food items taken from the smart refrigerator;

[0007] The target image is dehazed to obtain the image to be processed, and the stored food ingredients in the image to be processed are identified; the stored food ingredients are the food ingredients that have been pre-stored in the smart refrigerator.

[0008] If the name of the stored food ingredient is not identified, the image to be processed is compared with the set of stored food ingredient images for similarity; wherein, the set of stored food ingredient images is a collection of stored food ingredient images, and each type of food ingredient stored in the smart refrigerator has a corresponding stored food ingredient image;

[0009] Based on the comparison results, the target food image corresponding to the image to be processed in the stored food image set is determined, and the food name of the stored food is determined based on the food in the target food image.

[0010] In this embodiment, after acquiring the target image, a dehazing process is performed on the target image to obtain the image to be processed. The stored food items in the image to be processed are then identified. If the name of a stored food item is not identified, a similarity comparison is performed between the image to be processed and the set of stored food item images. In the above process, the ability to identify stored food items is improved by first identifying the dehazed image to be processed. Since the set of stored food item images is a collection of images corresponding to each type of food item stored in the smart refrigerator, when the dehazed stored food item cannot be identified, a similarity comparison between the image to be processed and the set of stored food item images can be used to obtain the target food item image that is closest to the image to be processed. Then, the name of the stored food item in the image to be processed is determined based on the food item corresponding to the target food item image, thereby reducing the food item misidentification rate.

[0011] In some possible embodiments, an image acquisition device is provided at a designated location of the smart refrigerator, and before acquiring the target image, the method further includes:

[0012] Determine if the user is currently performing a retrieval operation;

[0013] The user's current action is determined in the following way:

[0014] If the refrigerator door is detected to be open, the image acquisition device is controlled to acquire multiple frames of images of a designated area within a preset time period.

[0015] The location region of the object of interest in each frame image is determined, and the movement trajectory of the object of interest in the multiple frames image is determined based on the location region in each frame image; wherein, the object of interest may include the user's hand and / or the food items accessed by the user;

[0016] Based on the movement trajectory, it is determined whether the user's current operation is a deposit or a retrieval operation.

[0017] In this embodiment, after detecting that the refrigerator door has been opened, the image acquisition device is controlled to acquire multiple frames of images of a designated area within a preset time period. The location region of the object of interest (OP) in each frame is determined. The OOP may include the user's hand information and / or the food items the user is storing or retrieving. Based on the location region of the OOP in each frame, the movement trajectory of the OOP in the multiple frames is determined, and then based on this movement trajectory, it is determined whether the user opened the refrigerator door for a storage or retrieval operation. In this embodiment, when the user is performing a retrieval operation, the target image is acquired, and the stored food items in the target image are identified to reduce the false recognition rate of food items.

[0018] In some possible embodiments, acquiring the target image includes:

[0019] The multi-frame images are scored using a pre-trained recognition model;

[0020] The image with the highest score is selected as the target image.

[0021] This application employs a pre-trained recognition model to score multiple frames of images. A higher image score indicates a greater likelihood that the image contains food ingredients. Using the image with the highest score as the target image maximizes the chances of the target image containing food ingredients.

[0022] In some possible embodiments, the dehazing process on the target image to obtain the image to be processed includes:

[0023] Determine the dark channel image corresponding to the target image;

[0024] The ambient light in the target image is determined based on the brightness value of each pixel in the dark channel image;

[0025] The image to be processed corresponding to the target image is determined based on the ambient light.

[0026] In this embodiment of the application, when dehazing a target image, the dark channel image corresponding to the target image is first determined, and then the ambient light in the target image is determined based on the brightness value of each pixel in the dark channel image. The target image is then dehazed using this ambient light to obtain the image to be processed.

[0027] In some possible embodiments, comparing the image to be processed with a stored set of food ingredient images includes:

[0028] For each stored food image, a hash algorithm is used to determine the Hamming distance between the stored food image and the image to be processed;

[0029] The step of determining the target food image corresponding to the image to be processed in the stored food image set based on the comparison result includes:

[0030] The image with the smallest Hamming distance between the stored food image set and the image to be processed is taken as the target food image.

[0031] This embodiment uses a hash algorithm to determine the Hamming distance between each stored food image and the image to be processed, and then selects the image with the smallest Hamming distance as the target food image. Since a smaller Hamming distance between two images indicates that the two images are more similar, the above method can find the image in the stored food image set that is most similar to the image to be processed.

[0032] In some possible embodiments, determining the Hamming distance between the stored food image and the image to be processed using a hash algorithm includes:

[0033] Reduce the stored food image and the image to be processed to a specified size;

[0034] A perceptual hash algorithm is used to determine the first differential hash value of the stored food image and the second differential hash value of the image to be processed;

[0035] The Hamming distance is determined based on the difference between the first difference hash value and the second difference hash value.

[0036] In this embodiment, the stored food image and the image to be processed are pre-reduced to a specified size so that the two images have the same number of rows and columns of pixels. Thus, by employing a perceptual hash algorithm to determine the difference hash value between the stored food image and the image to be processed, and by determining the Hamming distance based on the difference hash value between the two images, the computation speed can be effectively improved.

[0037] In some possible embodiments, each type of food item stored in the smart refrigerator has a corresponding access record; after identifying the stored food items in the image to be processed, the method further includes:

[0038] If the name of the stored ingredient is identified, a retrieval record is added to the stored ingredient based on the image to be processed.

[0039] In the smart refrigerator of this application embodiment, a storage and retrieval record is set for each type of stored food. After recognizing the name of the stored food in the image to be processed, a corresponding retrieval record can be added to the stored food so that the user can check the food in the smart refrigerator at any time.

[0040] Secondly, embodiments of this application provide a food identification device for a smart refrigerator, the device comprising:

[0041] The image acquisition module is configured to acquire a target image containing stored food items taken from the smart refrigerator;

[0042] The dehazing module is configured to perform dehazing processing on the target image to obtain an image to be processed, and to identify stored food ingredients in the image to be processed; the stored food ingredients are food ingredients pre-stored in the smart refrigerator;

[0043] The similarity comparison module is configured to perform a similarity comparison between the image to be processed and the stored food image set if the food name of the stored food is not identified; wherein, the stored food image set is a collection of stored food images, and each type of food stored in the smart refrigerator has a corresponding stored food image;

[0044] The ingredient confirmation module is configured to determine the target ingredient image corresponding to the image to be processed in the stored ingredient image set based on the comparison result, and to determine the ingredient name of the stored ingredient based on the ingredients in the target ingredient image.

[0045] In some possible embodiments, an image acquisition device is provided at a designated location of the smart refrigerator, and before acquiring the target image, the image acquisition module is further configured to:

[0046] Determine if the user is currently performing a retrieval operation;

[0047] The user's current action is determined in the following way:

[0048] If the refrigerator door is detected to be open, the image acquisition device is controlled to acquire multiple frames of images of a designated area within a preset time period.

[0049] The location region of the object of interest in each frame image is determined, and the movement trajectory of the object of interest in the multiple frames image is determined based on the location region in each frame image; wherein, the object of interest may include the user's hand and / or food items accessed by the user;

[0050] Based on the movement trajectory, it is determined whether the user's current operation is a deposit or a retrieval operation.

[0051] In some possible embodiments, the acquisition of the target image is performed, and the image acquisition module is configured to:

[0052] The multi-frame images are scored using a pre-trained recognition model;

[0053] The image with the highest score is selected as the target image.

[0054] In some possible embodiments, the dehazing process on the target image is performed to obtain the image to be processed, and the dehazing module is configured to:

[0055] Determine the dark channel image corresponding to the target image;

[0056] The ambient light in the target image is determined based on the brightness value of each pixel in the dark channel image;

[0057] The image to be processed corresponding to the target image is determined based on the ambient light.

[0058] In some possible embodiments, the similarity comparison between the image to be processed and the stored set of food images is performed, and the similarity comparison module is configured to:

[0059] For each stored food image, a hash algorithm is used to determine the Hamming distance between the stored food image and the image to be processed;

[0060] The process of determining the target food image corresponding to the image to be processed in the stored food image set based on the comparison result is performed. The food determination module is configured as follows:

[0061] The image with the smallest Hamming distance between the stored food image set and the image to be processed is taken as the target food image.

[0062] In some possible embodiments, the similarity comparison module is configured to perform the determination of the Hamming distance between the stored food image and the image to be processed using a hash algorithm, wherein the similarity comparison module is configured to:

[0063] Reduce the stored food image and the image to be processed to a specified size;

[0064] A perceptual hash algorithm is used to determine the first differential hash value of the stored food image and the second differential hash value of the image to be processed;

[0065] The Hamming distance is determined based on the difference between the first difference hash value and the second difference hash value.

[0066] In some possible embodiments, each type of food stored in the smart refrigerator has a corresponding access record; after performing the identification of the stored food in the image to be processed, the food confirmation module is further configured to:

[0067] If the name of the stored ingredient is identified, a retrieval record is added to the stored ingredient based on the image to be processed.

[0068] Thirdly, embodiments of this application also provide an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method provided in the first aspect of embodiments of this application.

[0069] Fourthly, embodiments of this application also provide a computer storage medium storing a computer program for causing a computer to execute the method provided in the first aspect of embodiments of this application.

[0070] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0071] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is an illustration of the effect of food being frozen at low temperature, as shown in an embodiment of this application.

[0073] Figure 2 This is an application scenario diagram illustrating an embodiment of this application;

[0074] Figure 3a A flowchart illustrating a food identification method for a smart refrigerator is shown in this embodiment of the application.

[0075] Figure 3b This is a schematic diagram of a camera module shown in an embodiment of this application;

[0076] Figure 3c This is a schematic diagram of the hand movement trajectory shown in an embodiment of this application;

[0077] Figure 3d This is a schematic diagram illustrating the smart refrigerator recognizing and storing food items, as shown in an embodiment of this application.

[0078] Figure 3e This is a flowchart illustrating the determination of the Hamming distance, as shown in an embodiment of this application.

[0079] Figure 3f This is a schematic diagram illustrating the addition of a new take-out record for the taken-out ingredients, as shown in an embodiment of this application.

[0080] Figure 4 This is a structural diagram of the food identification device 400 for a smart refrigerator shown in an embodiment of this application;

[0081] Figure 5 This is a schematic diagram of an electronic device shown in an embodiment of this application. Detailed Implementation

[0082] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " will mean "or", for example, A / B can mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0083] In the description of the embodiments of this application, unless otherwise stated, the term "multiple" refers to two or more, and other quantifiers are similarly understood. The preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0084] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the control device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0085] Precise management of food information is a core function of smart refrigerators. In practical applications, accurate identification of food types is a crucial prerequisite for achieving precise food information management. Currently, in the field of smart refrigerators, food type identification and entry are mainly achieved through visual recognition using the refrigerator's built-in camera. This technology is implemented through two main methods: static recognition and dynamic recognition.

[0086] Static recognition involves using a built-in camera to capture images of the refrigerator's interior after the user closes the door, identifying the types of food inside. The effectiveness of this method is greatly influenced by the food's placement within the refrigerator, and it's prone to missing items due to obstruction between items. Dynamic recognition, on the other hand, uses the refrigerator's camera to capture at least one frame of the food image as the user accesses it. Because dynamic recognition minimizes obstruction and can provide guidance or reminders based on the recognition results, such as recommending storage locations and methods, most smart refrigerators currently employ dynamic recognition technology. However, after food has been stored in a smart refrigerator for a period of time, the low temperature can cause it to fog up, frost up, or deform due to freezing.

[0087] Specifically, such as Figure 1 As shown, Figure 1 The left side shows an image of a bag of frozen chicken immediately after it was placed in the smart refrigerator. The image shows the bag of frozen chicken removed after being placed in the freezer compartment for a period of time. Figure 1 As shown on the right. From Figure 1 As can be seen in the image on the right, the packaging of the bag of frozen chicken is covered with a large amount of frost and has undergone slight deformation due to the low temperature. In this case, a smart refrigerator may not be able to detect the contents of the package from the image. Figure 1 In the image on the right, frozen chicken is identified. For example, a smart refrigerator might mistake frozen chicken for similar ingredients such as pork or lamb, or it might not recognize any food in the image at all. Therefore, all of the above situations can lead to incorrect identification of food.

[0088] To address the aforementioned problems, the inventive concept of this application is as follows: After acquiring a target image, the embodiments of this application perform dehazing processing on the target image to obtain an image to be processed. Stored food items in the image to be processed are then identified. If the name of a stored food item is not identified, a similarity comparison is performed between the image to be processed and the set of stored food item images. In the above process, the ability to identify stored food items is first improved by identifying the dehazed image to be processed. Since the set of stored food item images is a collection of images corresponding to each type of food item stored in the smart refrigerator, when the dehazed stored food item cannot be identified, a similarity comparison between the image to be processed and the set of stored food item images can be used to obtain the target food item image most similar to the image to be processed. Then, the name of the stored food item in the image to be processed is determined based on the food item corresponding to the target food item image, thereby reducing the food item misidentification rate.

[0089] Specifically, such as Figure 2 The diagram shown is an application scenario diagram of a food identification method for a smart refrigerator provided in this application embodiment. The application scenario includes a user terminal 10, a smart refrigerator 20, and a server 30.

[0090] The user terminal 10 can be an electronic device with communication capabilities, such as a smartphone, a PDA, or a tablet computer. The server 30 can be a built-in server of the smart refrigerator 20, a standalone server cluster, or a cloud server.

[0091] In practical applications, an application for viewing the food inside the smart refrigerator 20 can be installed on the user terminal 10. Each time the user opens the refrigerator door of the smart refrigerator 20, the smart refrigerator 20 controls the image acquisition device to capture multiple frames of images from a specified location within a specified time period and sends these images to the server 30. The server 30 determines whether the user is storing or retrieving food by recognizing the user's hand movement trajectory in the multiple frames of images.

[0092] In some possible embodiments, if it is determined that the user is retrieving food, a target image is selected from multiple frames, the name of the food in the target image is identified, and then a record of the food retrieval is made. This retrieval record is then sent to the user terminal 10 so that the user can be informed of the food status inside the smart refrigerator 20 in a timely manner.

[0093] It should also be noted that the user terminal 10, the smart refrigerator 20, and the server 30 can be connected and transmit data via wired or wireless connections. For example, the user terminal 10, the smart refrigerator 20, and the server 30 can be connected via a data cable or a wired network. Furthermore, the user terminal 10, the smart refrigerator 20, and the server 30 can also be connected via an RF module, a Bluetooth module, or a wireless network; this application does not limit the connection in this regard.

[0094] To facilitate understanding of the food identification method for a smart refrigerator provided in this application embodiment, the specific details are as follows: Figure 3a As shown, it includes the following steps:

[0095] Step 301: Obtain a target image, the target image containing stored food items taken from the smart refrigerator;

[0096] The smart refrigerator of this embodiment has an image acquisition device at a designated location. When it is determined that a user is performing a take-out operation, the image acquisition device can be controlled to acquire multiple frames of images of the designated area within a preset time period. Specifically, a camera module can be installed above the smart refrigerator. This camera module may include an RGB camera and a depth camera. The RGB camera is used to acquire color images, while the depth camera is used to obtain depth information from the images. Figure 3b As shown, the camera module can be installed above the smart refrigerator for shooting from a top-down angle. Alternatively, the camera module can be placed inside the freezer and automatically pop out when the refrigerator door is opened; however, this embodiment does not limit this option.

[0097] The smart refrigerator can be equipped with a sensor on the refrigerator door. When the door is detected to be open, the sensor controls the image acquisition device to capture multiple frames of images of a designated area within a preset time period. Then, based on the hand position in each frame, the sensor determines the corresponding hand movement trajectory and identifies whether the user is currently performing a storage or retrieval operation. The designated area is the region where the user's hand image can be completely captured.

[0098] In practice, multiple frames of images are continuously captured from a designated area within a preset time period (e.g., 1 minute). For example, after acquiring 30 frames, the object of interest (OPI) is determined in each frame. OPI may include the user's hands and / or the food items the user is storing or retrieving. By determining the movement trajectory of the OPI in these 30 frames based on the time sequence of the captured images, it can be determined whether the user opened the refrigerator door to store or retrieve food items.

[0099] Taking the user's hand as the object of interest as an example, hand position detection can be performed on any frame of the image. The detection result includes the coordinates of the hand in the image (such as the coordinates in the camera's two-dimensional image coordinate system). After determining the hand coordinates, a target tracking algorithm can be used to track the hand position region in each frame of the image, thereby determining the hand movement trajectory across 30 frames. Specifically... Figure 3c As shown, when the movement trajectory of the hand position area in the aforementioned 30 frames indicates that the user's hand is moving from inside the refrigerator's outer compartment, it is determined that the user is currently performing a storage operation. Conversely, when the movement trajectory indicates that the user's hand is moving from inside the refrigerator to outside, it is determined that the user is currently performing a retrieval operation.

[0100] As mentioned earlier, smart refrigerators have the function of recording the food stored inside. This is because smart refrigerators have the ability to recognize common food items on the market. The technology primarily achieves this by using a recognition model to identify images captured by an image acquisition device. However, the low temperature inside the refrigerator can cause food to fog up, frost up, or deform due to freezing, all of which can prevent the smart refrigerator from correctly identifying the food. In this embodiment, after determining that the user has performed a take-out operation, a pre-trained recognition model scores the aforementioned multiple frames of images. The higher the score of the recognition model, the greater the likelihood that the image contains food. Therefore, when determining the target image, the image with the highest score should be selected as the target image to improve the recognition rate of food in the target image.

[0101] Step 302: Perform dehazing processing on the target image to obtain the image to be processed, and identify the stored food ingredients in the image to be processed; the stored food ingredients are the food ingredients pre-stored in the smart refrigerator;

[0102] After acquiring the target image, dehazing is performed on the target image to obtain the image to be processed. In practice, the dark channel image corresponding to the target image is first determined, then the brightness value of each pixel in the dark channel image is obtained, and the ambient light in the target image is determined based on the brightness value of each pixel in the dark channel image. Finally, the image to be processed corresponding to the target image is determined based on the ambient light.

[0103] Specifically, the dark channel image corresponding to the target image is first determined using the following formula (1):

[0104]

[0105] Among them, J dark (x) represents the pixel with the smallest pixel value in the dark channel; Ω(x) represents a window centered at pixel x, y represents the pixels within that window, the superscript c represents the R, G, and B color channels, and J... c (y) represents the pixel values ​​of the three channels in this window.

[0106] The fog image imaging model in computer vision and computer graphics is as follows (2):

[0107] I(x)=J(x)t(x)+A(1-t(x)) formula (2)

[0108] Where I(x) is the target image, J(x) is the image to be processed after dehazing, A is the ambient light, and t(x) is the transmittance. Now that I(x) is known, we need to calculate J(x) in equation (2). In practice, we can divide both sides of equation (2) by the ambient light to obtain the following equation (3):

[0109]

[0110] Let the transmittance t(x) within each window be a constant, and define it as: Then, by taking the minimum value twice on both sides of equation (3), we can obtain the following formula (4):

[0111]

[0112] Furthermore, J in formula (1) dark When the value of (x) is determined to be 0, the following formula (5) is obtained:

[0113]

[0114] Among them, the above It can be determined by the following formula (6).

[0115]

[0116] It should also be noted that the above calculation method for ambient light A is as follows: Based on the pixel brightness values, the top 0.1% of pixels with the highest brightness values ​​are selected from the dark channel image. Then, based on the positions of these pixels in the dark channel image, the pixels at the same positions in the target image are determined. Finally, the pixel with the highest brightness value among all pixels at the same positions in the target image is used as the value of ambient light A. Specifically, it can be shown in the following formula (7):

[0117]

[0118] After determining the image to be processed through the above process, the stored food items in the image are identified. As mentioned earlier, the smart refrigerator captures images every time a user stores or retrieves food items. Stored food items refer to those pre-stored in the smart refrigerator. This application embodiment constructs a set of stored food item images for each stored item. That is, each time a user performs a storage operation, the name of the stored food item and the image captured at the time of the storage operation are entered.

[0119] Specifically, it can be as follows Figure 3d As shown, for example, if a user opens the refrigerator door and the system detects that the user has performed a storage operation, then it identifies the stored food items. Figure 3d The image shown is a green apple. The image is then associated with the green apple. Through this process, when a user stores food items already in the smart refrigerator, an associated image can be added under that food item's name. For example, if there were no eggs in the refrigerator initially, and the user stores one in the morning, the smart refrigerator, upon recognizing the egg, will list it as a stored food item and associate the image of the egg stored that morning with the food item name "egg." If, in the afternoon of the same day, the user stores another egg, the smart refrigerator will again associate the image of the egg stored that afternoon with the food item name "egg," resulting in a total of two images associated with the food item name "egg." Thus, a stored food item image set can be created for the food items already stored in the smart refrigerator.

[0120] Step 303: If the name of the stored food ingredient is not identified, the image to be processed is compared with the set of stored food ingredient images for similarity; wherein, the set of stored food ingredient images is a collection of stored food ingredient images, and each type of food ingredient stored in the smart refrigerator has a corresponding stored food ingredient image;

[0121] If the smart refrigerator fails to recognize the name of the stored food item, there are two possibilities. One is that the smart refrigerator mistakenly identifies the food item in the image as something else entirely, and this food item was not originally stored in the refrigerator, therefore its name is not among the stored items. The other possibility is that the smart refrigerator simply cannot recognize the food item in the image. In either case, it indicates that the smart refrigerator cannot correctly identify what food the user has just taken out.

[0122] To address the aforementioned issues, this application embodiment compares the image to be processed with a stored set of food ingredient images for similarity. Specifically, for each stored food ingredient image, a hash algorithm is used to determine the Hamming distance between the stored food ingredient image and the image to be processed.

[0123] The process of calculating the Hamming distance can be as follows: Figure 3e As shown, the first step is to reduce the size. Specifically, the stored food image and the image to be processed are reduced to a specified size. In this embodiment, the stored food image and the image to be processed are reduced to a size of 9*8, that is, each image has 64 pixels. This eliminates the differences in images caused by different sizes and proportions, and improves the calculation speed.

[0124] Then, step B is performed: grayscale processing. In this embodiment, a perceptual hash algorithm is used to determine the first difference hash value of the stored food image and the second difference hash value of the image to be processed, and the Hamming distance is determined based on the difference between the first and second difference hash values. The perceptual hash algorithm derives the difference hash value by calculating the color intensity difference between adjacent pixels. Directly comparing color intensity differences using RGB values ​​is quite complex, so the RGB values ​​can be converted to grayscale values ​​first, represented by a single integer from 0 to 255. This simplifies the three-dimensional comparison to a one-dimensional comparison, thereby improving the calculation speed.

[0125] Next, we proceed to step C: difference calculation. The difference value is obtained by calculating the intensity contrast between adjacent pixels in each row. In step A, the stored food image and the image to be processed were reduced to a resolution of 9*8. That is, each image has 8 rows of pixels, and each row has 9 pixels. The difference value is calculated for each row separately, meaning that the first pixel of the second row will not be compared with any pixel in the first row. Since each row has 9 pixels, there will be 8 difference values. Because 8 bits can form a byte, it is easy to convert to a hexadecimal value. This is why 9 was chosen as the image width in step A.

[0126] Step D: Hash Value Conversion. It should be noted that during the difference calculation in Step C above, assuming pixel A and pixel B are two adjacent pixels in the same row, and pixel A is ordered before pixel B, if the color intensity of pixel A is greater than that of pixel B, the difference value between the two pixels is 1; otherwise, it is 0. Therefore, for each image (storing the food image and the image to be processed), a 64-bit difference value array can be obtained, where each bit in the array represents the difference value between adjacent pixels. Thus, each value in the difference value array is considered as one bit, and every 8 bits are grouped into a hexadecimal value. Concatenating these hexadecimal values ​​and converting them into a string yields the difference hash (dHash) of the image corresponding to that difference value array.

[0127] Step E: Determine the Hamming distance. The Hamming distance in dHash is calculated by determining the number of bits modified in the difference value. Since the difference value in this embodiment is represented by 0 and 1 (i.e., each bit in the difference value array), it can be treated as binary, and the number of bits that differ in the result is calculated; this is the Hamming distance. This effectively improves the calculation speed. The smaller the Hamming distance between two images, the higher the probability that the two images are the same. Generally, two images with a Hamming distance of no more than 10 have essentially the same content.

[0128] Step 304: Determine the target food image corresponding to the image to be processed in the stored food image set according to the comparison results, and determine the food name of the stored food based on the food in the target food image.

[0129] After determining the Hamming distance between each stored food image in the stored food image set and the image to be processed through steps A to E above, the stored food image with the smallest Hamming distance to the image to be processed can be selected as the target food image. That is, at this point, the food in the target food image is considered to be the food in the image to be processed.

[0130] In the smart refrigerator of this application embodiment, a storage and retrieval record is set for each type of stored food. That is, every time the smart refrigerator detects that a user performs a storage or retrieval operation, it will identify what food the user has stored or retrieved and add a corresponding record for that food.

[0131] Specifically, it can be as follows Figure 3f As shown, suppose the smart refrigerator already stores the following ingredients: 2 heads of cabbage, 3 tomatoes, and 12 eggs. User A stored 10 eggs on month 1 and 2 eggs on month 2. If user A takes out 3 eggs on month 5, when the smart refrigerator detects that the image to be processed includes 3 eggs, it indicates that the user opened the refrigerator door and took out 3 eggs. At this point, a corresponding withdrawal record can be added for the stored ingredient, the eggs.

[0132] The above process compares the image to be processed with the stored set of food images to determine the target food image, and then determines the food name of the food already stored in the image to be processed based on the target food image, thereby reducing the misidentification rate of food by the smart refrigerator.

[0133] Based on the same inventive concept, this application also provides a food identification device 400 for a smart refrigerator, specifically as follows: Figure 4 As shown, it includes:

[0134] Image acquisition module 401 is configured to acquire a target image containing stored food items taken from a smart refrigerator;

[0135] The dehazing module 402 is configured to perform dehazing processing on the target image to obtain an image to be processed, and to identify stored food ingredients in the image to be processed; the stored food ingredients are food ingredients pre-stored in the smart refrigerator;

[0136] The similarity comparison module 403 is configured to perform a similarity comparison between the image to be processed and the stored food image set if the food name of the stored food is not identified; wherein, the stored food image set is a collection of stored food images, and each type of food stored in the smart refrigerator has a corresponding stored food image;

[0137] The ingredient confirmation module 404 is configured to perform the following actions: determine the target ingredient image corresponding to the image to be processed in the stored ingredient image set based on the comparison result, and determine the ingredient name of the stored ingredient based on the ingredients in the target ingredient image.

[0138] In some possible embodiments, an image acquisition device is provided at a designated location of the smart refrigerator, and before acquiring the target image, the image acquisition module 401 is further configured to:

[0139] Determine if the user is currently performing a retrieval operation;

[0140] The user's current action is determined in the following way:

[0141] If the refrigerator door is detected to be open, the image acquisition device is controlled to acquire multiple frames of images of a designated area within a preset time period.

[0142] The location region of the object of interest in each frame image is determined, and the movement trajectory of the object of interest in the multiple frames image is determined based on the location region in each frame image; wherein, the object of interest may include the user's hand and / or food items accessed by the user;

[0143] Based on the movement trajectory, it is determined whether the user's current operation is a deposit or a retrieval operation.

[0144] In some possible embodiments, the acquisition of the target image is performed, and the image acquisition module 401 is configured to:

[0145] The multi-frame images are scored using a pre-trained recognition model;

[0146] The image with the highest score is selected as the target image.

[0147] In some possible embodiments, the dehazing process on the target image is performed to obtain the image to be processed, and the dehazing module 402 is configured to:

[0148] Determine the dark channel image corresponding to the target image;

[0149] The ambient light in the target image is determined based on the brightness value of each pixel in the dark channel image;

[0150] The image to be processed corresponding to the target image is determined based on the ambient light.

[0151] In some possible embodiments, the similarity comparison module 403 is configured to perform the comparison between the image to be processed and the stored set of food images.

[0152] For each stored food image, a hash algorithm is used to determine the Hamming distance between the stored food image and the image to be processed;

[0153] The process of determining the target food image corresponding to the image to be processed in the stored food image set based on the comparison result is performed. The food determination module is configured as follows:

[0154] The image with the smallest Hamming distance between the stored food image set and the image to be processed is taken as the target food image.

[0155] In some possible embodiments, the similarity comparison module 403 is configured to perform the determination of the Hamming distance between the stored food image and the image to be processed using a hash algorithm, wherein the similarity comparison module 403 is configured to:

[0156] Reduce the stored food image and the image to be processed to a specified size;

[0157] A perceptual hash algorithm is used to determine the first differential hash value of the stored food image and the second differential hash value of the image to be processed;

[0158] The Hamming distance is determined based on the difference between the first difference hash value and the second difference hash value.

[0159] In some possible embodiments, each type of food stored in the smart refrigerator has a corresponding access record; after performing the identification of the stored food in the image to be processed, the food confirmation module 404 is further configured to:

[0160] If the name of the stored ingredient is identified, a retrieval record is added to the stored ingredient based on the image to be processed.

[0161] The following reference Figure 5 To describe an electronic device 130 according to this embodiment of the present application. Figure 5 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0162] like Figure 5 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0163] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0164] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0165] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0166] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0167] In some possible implementations, various aspects of the food identification method for a smart refrigerator provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the food identification method for a smart refrigerator according to the various exemplary embodiments of this application described above.

[0168] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0169] The program product for food identification in a smart refrigerator according to embodiments of this application can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0170] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0171] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0172] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0173] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0174] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0175] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] This application is described with reference to flowchart illustrations and block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block and / or block in the flowchart illustrations and block diagrams, as well as combinations of blocks and processes in the flowchart illustrations and block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0177] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and boxes Figure 1 The steps of the function specified in one or more boxes.

[0179] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0180] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for identifying food items in a smart refrigerator, characterized in that, The method includes: Acquire a target image, the target image containing stored food items taken from a smart refrigerator; From the dark channel image corresponding to the target image, at least one dark channel pixel with a brightness value higher than a preset threshold is selected; wherein, each dark channel pixel corresponds to a target pixel, each target pixel is located in the target image, and has the same coordinates as the corresponding dark channel pixel; The maximum brightness value of each target pixel is taken as the ambient light, and the target image is dehazed based on the ambient light and the transmittance of the target image to obtain the image to be processed. If the name of a stored ingredient in the image to be processed is not identified, the Hamming distance between each stored ingredient image and the image to be processed is determined in the following way: The stored food image and the image to be processed are reduced to a preset size; and the reduced-size stored food image and the image to be processed are converted into their respective grayscale images. For each grayscale image, the following steps are performed on each row of pixels in the grayscale image: according to the arrangement order of the pixels in the row, the grayscale values ​​of every two adjacent pixels in the current row are compared and a one-bit binary number is generated to describe the comparison result; and the binary numbers are concatenated in the order they are generated to obtain a multi-bit binary number corresponding to the current row. For each grayscale image, the corresponding binary numbers of each row are concatenated sequentially according to the row arrangement order to obtain the concatenation result of the binary numbers corresponding to each grayscale image. The Hamming distance is determined based on the difference between the binary concatenation results of the two grayscale images; wherein, the stored food image set is a collection of stored food images, and each type of food stored in the smart refrigerator has a corresponding stored food image; The image with the smallest Hamming distance between the stored food image set and the image to be processed is taken as the target food image, and the food name of the stored food is determined based on the food in the target food image.

2. The method according to claim 1, characterized in that, The smart refrigerator is equipped with an image acquisition device at a designated location. Before acquiring the target image, the method further includes: Determine if the user is currently performing a retrieval operation; The user's current action is determined in the following way: If the refrigerator door is detected to be open, the image acquisition device is controlled to acquire multiple frames of images of a designated area within a preset time period. The location region of the object of interest in each frame image is determined, and the movement trajectory of the object of interest in the multiple frames image is determined based on the location region in each frame image; wherein, the object of interest includes the user's hand and / or food items accessed by the user; Based on the movement trajectory, it is determined whether the user's current operation is a deposit or a retrieval operation.

3. The method according to claim 2, characterized in that, The acquisition of the target image includes: The multi-frame images are scored using a pre-trained recognition model; The image with the highest score is selected as the target image.

4. The method according to any one of claims 1-3, characterized in that, Each type of food item stored in the smart refrigerator has a corresponding access record; after identifying the stored food items in the image to be processed, the method further includes: If the name of the stored ingredient is identified, a retrieval record is added to the stored ingredient based on the image to be processed.

5. A food identification device for a smart refrigerator, characterized in that, The device includes: The image acquisition module is configured to acquire a target image containing stored food items taken from the smart refrigerator; The dehazing module is configured to select at least one dark channel pixel with a brightness value higher than a preset threshold from the dark channel image corresponding to the target image; wherein each dark channel pixel corresponds to a target pixel, each target pixel is located in the target image, and has the same coordinates as the corresponding dark channel pixel. The maximum brightness value of each target pixel is taken as the ambient light, and the target image is dehazed based on the ambient light to obtain the image to be processed. The similarity comparison module is configured to determine the Hamming distance between each stored food ingredient image and the image to be processed if no food ingredient name is identified in the image to be processed, in the following manner: The stored food image and the image to be processed are reduced to a preset size; and the reduced-size stored food image and the image to be processed are converted into their respective grayscale images. For each grayscale image, the following steps are performed on each row of pixels in the grayscale image: according to the arrangement order of the pixels in the row, the grayscale values ​​of every two adjacent pixels in the current row are compared and a one-bit binary number is generated to describe the comparison result; and the binary numbers are concatenated in the order they are generated to obtain a multi-bit binary number corresponding to the current row. For each grayscale image, the corresponding binary numbers of each row are concatenated sequentially according to the row arrangement order to obtain the concatenation result of the binary numbers corresponding to each grayscale image. The Hamming distance is determined based on the difference between the binary data of the two grayscale images; wherein, the stored food image set is a collection of stored food images, and each type of food stored in the smart refrigerator has a corresponding stored food image; The ingredient confirmation module is configured to select the image with the smallest Hamming distance from the image to be processed in the stored ingredient image set as the target ingredient image, and determine the ingredient name of the stored ingredient based on the ingredients in the target ingredient image.

6. An electronic device, characterized in that, The method includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-4.

7. A computer storage medium, characterized in that, The computer storage medium stores a computer program that enables the computer to perform the method as described in any one of claims 1-4.

Images