A cabinet food identification system and identification method

By designing a cabinet and cabinet food recognition system, using remote cloud servers to process image information collected by cameras, the problem of low food recognition rate in smart homes is solved, and efficient and low-cost food management and further intelligence of smart homes is achieved.

CN111950414BActive Publication Date: 2024-11-22GUANGZHOU WEILIN SOFTWARE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010759222.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-31
Publication Date
2024-11-22
Estimated Expiration
2040-07-31

AI Technical Summary

Technical Problem

The food recognition system in existing smart homes has low recognition rate and is expensive, and lacks effective food management functions.

Method used

A cabinet food identification system was designed, including identification mechanism, video streaming server, identification server and storage server. Image information is collected through cameras, image recognition and data processing is used for remote cloud servers, food types, inlet and exit times, and Update data information in the storage server.

Benefits of technology

It realizes high-recognition and low-cost food management, can identify different types of items, and record food types, cabinet IDs, food characteristics, and time for putting in and taking out, improving the level of home intelligence.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111950414B_ABST
    Figure CN111950414B_ABST
Patent Text Reader

Abstract

The present invention discloses a cabinet food identification system and identification method, comprising an identification mechanism, a video stream server, an identification server and a storage server, wherein: the identification mechanism is used to collect image information of food entering and leaving the cabinet, and transmit it to the video stream server; the video stream server receives the image information collected by the identification mechanism, and distributes the image information to the identification server; the identification server receives the image information, identifies the picture according to the image information, and performs tracking judgment and identification judgment through the picture, and finally sends the judgment result to the storage server; the storage server receives the judgment result and updates the food data information in the cabinet; the cabinet food identification system and identification method have a high degree of intelligence and a good recognition effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention specifically relates to a cabinet food identification system and an identification method. Background Art

[0002] Smart home (smart home, home automation) is based on the house as a platform, using integrated wiring technology, network communication technology, security technology, automatic control technology, audio and video technology to integrate facilities related to home life, build an efficient management system for residential facilities and family schedule affairs, improve home safety, convenience, comfort, artistry, and achieve an environmentally friendly and energy-saving living environment.

[0003] In recent years, with the popularity of smart homes, more and more household appliances have been upgraded to intelligent ones. In existing smart homes, refrigerators, cabinets, food cabinets or storage cabinets used to store food, the identification and management of food is one of the key points. Conventional smart homes use built-in cameras with limited shooting field of view, low recognition rate, and overall high price. Many of them do not even have recognition functions. Summary of the invention

[0004] In view of the deficiencies in the prior art, an object of the present invention is to provide a cabinet food recognition system and recognition method with a high degree of intelligence and good recognition effect.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A cabinet food identification system includes an identification mechanism, a video stream server, an identification server and a storage server, wherein:

[0007] The identification mechanism is used to collect image information of food entering and leaving the cabinet and transmit it to the video streaming server;

[0008] The video stream server receives the image information collected by the recognition mechanism and distributes the image information to the recognition server;

[0009] The recognition server receives the image information, recognizes the picture according to the image information, performs tracking judgment and recognition judgment through the picture, and finally sends the judgment result to the storage server;

[0010] The storage server receives the judgment result and updates the food data information in the cabinet.

[0011] Preferably, the cabinet is one of a cabinet, a refrigerator, a food cabinet, and a storage cabinet.

[0012] Furthermore, the identification mechanism, video stream server, identification server and storage server are all remote cloud servers, and data is transmitted between each other through internal and external means.

[0013] Furthermore, the identification mechanism includes at least one camera disposed outside the box and located on the top of the box.

[0014] Preferably, the identification mechanism includes at least one camera disposed in the box and located at the bottom of the box.

[0015] Preferably, the food data information includes food type, cabinet ID, food characteristics, time of putting in and time of taking out.

[0016] Another technical problem to be solved by the present invention is to provide a cabinet food identification method, which uses the cabinet food identification system as described above and includes the following steps:

[0017] 1) Acquire images by aiming the camera at the area in front of the cabinet door to capture the scene of items being put into or taken out of the cabinet. There is at least one camera, and each camera has an ID, which is bound to the cabinet and connected to the remote server;

[0018] 2) Transmit and distribute images. The images obtained by each cabinet need to be uploaded to the video streaming server, which will distribute them to the recognition server;

[0019] 3) Identify the food. The recognition model loaded by the recognition server identifies the corresponding picture. If the target is found, the ID and picture information of the current cabinet are retained and the food tracking mode is turned on.

[0020] 4) Tracking food, judging the state of food by the movement trend of food in multiple consecutive frames. When the model recognizes that the cabinet door is open, if the food is close to the cabinet, it can be judged as food being placed, and if the food is far away from the cabinet, it can be judged as food being taken;

[0021] 5) Manage food, identify the server, and after obtaining the food status, give feedback according to different statuses. When it is judged to put food, send the relevant data of the food to the storage server, store and update the cabinet food information, and send the result to the terminal for display at the same time; when it is judged to take food, send the relevant data of the food to the storage server, receive the characteristics of the same type of food returned by the storage server, compare the same type of food to screen out the most likely food, return it to the storage server, store and update the food information, and send the result to the terminal for display.

[0022] Preferably, in step 4), the food tracking method is:

[0023] The direction of food entry and exit is determined by the location of the camera. Let P(x, y) represent the coordinates of the food in the camera, P pre (x, y) represents the food coordinates of the previous frame, P now (x, y) represents the food coordinates in the current frame, P diff (x, y) represents the difference between the two. The specific formula is as follows: diff (x, y) = P now (x, y)-P pre (x, y), if it is close to the camera, it is considered to be food, then P diff (x, y) is positive, and moving away from the camera is considered to be taking food. diff (x, y) value is negative; when the coordinates P of the food in N consecutive frames diff (x, y) are both positive, and N is the algorithm preset value, then it is considered that the state at this time is to place food. When the food coordinates P in N consecutive frames diff If (x, y) is negative, it is considered that the current state is to take food.

[0024] Furthermore, in step 5), the smart terminal includes a mobile phone APP, a WeChat applet, and a display screen of the cabinet.

[0025] The beneficial effects of the present invention are:

[0026] The device is simple, inexpensive, and has a wide range of applications. It can be deployed on cabinets where food is stored. It is calculated through a dedicated cloud server with powerful computing power and can identify different types of items with good recognition effect. After recognition, it can store food type, cabinet ID, food characteristics, placement time, and removal time data in the storage server, thereby realizing further intelligence of the home. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the installation of a device in a cabinet food identification system and identification method of the present invention;

[0028] Figure 2 It is a schematic diagram of a server module in a cabinet food identification system and identification method of the present invention;

[0029] Figure 3 It is a schematic diagram of the overall process of a cabinet food identification system and identification method of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0031] Example

[0032] The image is obtained mainly through the camera installed on the refrigerator. Figure 1 As shown in the device installation diagram, camera 1 and camera 2 are installed on the top and front of the refrigerator. The purpose is to capture the food entering and leaving the refrigerator when the refrigerator door is opened. This example uses two cameras. In actual installation, the number and position of cameras installed will vary according to the size of the refrigerator and the installation environment.

[0033] Image transmission and distribution,Before the image is transmitted, the camera ID and the refrigerator ID need to be bound together.,In this example, all cameras are network cameras that can connect to the remote server.,During initialization, the server binds the connected camera ID to the refrigerator,,and each subsequent connection can confirm which refrigerator the data comes from.

[0034] In this example, the image is assigned as follows Figure 2 As shown in the figure, the camera bound to the refrigerator uploads the image data to the video streaming server through the network. The relationship between the camera and the video streaming server is N to 1, and each video streaming server receives the data uploaded by N cameras. At the same time, the video streaming server forwards the processed image data to the recognition server. The relationship between the video streaming server and the recognition server is 1 to N. As an intermediate server, the video streaming server connects to the terminal camera on the one hand, caches and preliminarily processes the image data transmitted by different cameras. On the other hand, it connects to the recognition server and distributes the image data to different recognition servers according to the processing progress of the recognition server.

[0035] like Figure 2 As shown in the figure, during the judgment process, the recognition server will exchange data with the storage server according to the processing situation. For example, when it is judged that food is taken, the corresponding food will be taken out from the storage server for comparison, and the comparison result will be returned to the storage server for storage. When it is judged that food is placed, data including food type, refrigerator ID, food characteristics, etc. will be sent to the storage server for storage. The corresponding relationship between the recognition server and the storage server is N to 1. One storage server can be connected to multiple recognition servers at the same time.

[0036] For food recognition, specifically, a pre-trained recognition model is loaded in the recognition server, and data transmitted from the video streaming server is received, including but not limited to the camera ID, image data, image timestamp, etc. If the image does not contain the target, the recognition is terminated. If the image contains the target, the current camera ID, image data, image timestamp, target area, type, credibility and other information are stored in the storage server, and the food tracking mode is turned on.

[0037] The main purpose of food tracking is to determine whether food is entering or leaving the refrigerator. The determination method in this example is to first determine the direction of food entering or leaving the refrigerator based on the location of the camera. Let P(x, y) represent the coordinates of the food in the camera, P pre (x, y) represents the food coordinates of the previous frame, P now (x, y) represents the food coordinates in the current frame, P diff (x, y) represents the difference between the two. The specific formula is as follows:

[0038] P diff (x, y) = P now (x, y)-P pre (x, y)

[0039] For example, if the camera is close to the food, P diff (x, y) is positive, and moving away from the camera is considered to be taking food. diff The (x, y) value is negative. The state of the food can be determined by the movement trend of the same type of food in multiple consecutive frames. For example, the coordinates P of the food in N consecutive frames diff (x, y) are both positive, and N is the algorithm preset value, then it is considered that the state at this time is to place food. If the food coordinates P in N consecutive frames diff If (x, y) is negative, then the state is considered to be taking food.

[0040] See also Figure 3 As shown, a method for identifying food in a cabinet comprises the following steps:

[0041] 1) Acquire images by aiming the camera at the area in front of the cabinet door to capture the scene of items being put into or taken out of the cabinet. There is at least one camera, and each camera has an ID, which is bound to the cabinet and connected to the remote server;

[0042] 2) Transmit and distribute images. The images obtained by each cabinet need to be uploaded to the video streaming server, which will distribute them to the recognition server;

[0043] 3) Identify the food. The recognition model loaded by the recognition server identifies the corresponding picture. If the target is found, the ID and picture information of the current cabinet are retained and the food tracking mode is turned on.

[0044] 4) Tracking food, judging the state of food by the movement trend of food in multiple consecutive frames. When the model recognizes that the cabinet door is open, if the food is close to the cabinet, it can be judged as food being placed, and if the food is far away from the cabinet, it can be judged as food being taken;

[0045] 5) Manage food, identify the server, and after obtaining the food status, give feedback according to different statuses. When it is judged to put food, send the relevant data of the food to the storage server, store and update the cabinet food information, and send the result to the terminal for display at the same time; when it is judged to take food, send the relevant data of the food to the storage server, receive the characteristics of the same type of food returned by the storage server, compare the same type of food to screen out the most likely food, return it to the storage server, store and update the food information, and send the result to the terminal for display.

[0046] The management of food mainly involves the data flow between several modules. Specifically, after the recognition server obtains the food status, different feedback is given according to different statuses. When it is judged that food is to be placed, data including food type, refrigerator ID, food characteristics, etc. are sent to the storage server, stored, and the refrigerator food information is updated, and the results are sent to the terminal for display at the same time. When it is judged that food is to be taken, the type of food, refrigerator ID and other data are sent to the storage server, and the characteristics of the same type of food returned by the storage server are received. The most likely food is compared and screened out for the same type of food, and returned to the storage server, stored, and updated. The food information is sent to the terminal for display at the same time. When the business volume is not large, the video streaming server and the storage server can be deployed in the same server. Specifically, the display terminal includes but is not limited to mobile phone APP, WeChat applets, and the display screen of the smart terminal refrigerator.

[0047] Among them, the camera acquisition system is as follows:

[0048] 1. Get camera video data:

[0049] When the camera detects that the object in front of the lens meets the trigger condition, it will send a signal to the video receiver, such as timed trigger, event (object movement) trigger, server soft trigger. After receiving the signal, the video receiver establishes a socket connection with the camera. After the connection is successfully established, the camera sends video data to the video receiver. The sent data includes video bytes, camera id (cameraId), and a unique id for each video (requestId: used to distinguish videos of the same camera in different time periods).

[0050] 2. Decode the video data

[0051] The video receiver sends the acquired video data to the video segmentation processor. Because the video data sent by the camera each time is urgent, the data received each time is not enough to synthesize a picture, so the video segmentation processor is needed to process the original data. The video segmentation processor will start a new thread after receiving a piece of data, using the camera id (cameraId) as an identifier, and continuously monitor the original video data acquired in the next minute. Every 3 seconds, this accumulated data will be decoded by h264 and captured. 10 frames (or so) of pictures will be captured per second. After the first successful capture of each video (distinguished by requestId), a new video image processor will be created and placed in the queue of the video image processor scheduling thread. Each captured picture is placed in the container of the video image processor. After the entire video is captured, the video image processor will be marked as captured.

[0052] 3. Process the video and pictures

[0053] When the server starts, a video image processor scheduling thread will be started. This thread will go back and continuously take out the video image processor from its own queue. When the video image processor is taken out, an image recognition interface caller will be used. The caller provides a function for calling the image recognition interface, which can recognize the image and return the image processing result (including the action of taking out or storing, the guessed category of food and other items, and the quantity of food and other items), and then assign the result to an image recognition result processor.

[0054] 4. Image recognition result processing

[0055] The image recognition result processor will store the obtained results in the cloud. For example, put 1 apple in refrigerator a, put 3 watermelons in refrigerator a, and take out 2 boxes of beef.

[0056] Among them, the video processing system is as follows:

[0057] The whole system is divided into imaging module, labeling module, training module, recognition module, trajectory tracking module and behavior determination module. The imaging module is placed in a fixed position through an external network camera to shoot and capture moving targets within the field of view.

[0058] The labeling module provides training data to the training module. It is divided into three parts. The first part is to label the palm, the second part is to label the hand-held object, and the third part is to label different types of objects.

[0059] The training module can select different models for training the labeled images. For example, the two-step rcnn series model or the one-step yolo series. The training categories include three categories: palm, palm-held objects, and objects. The first two categories are fixed, and the last category of objects is selected according to actual needs. It can be one object or multiple objects.

[0060] The recognition module uses the trained model to recognize the image. The objects include palm A, palm-held object B, and held object C. It gives the recognized target location box, category class, and score. The reason for recognizing the three objects is that the handheld object itself is easily blocked and has a low recognition rate. The palm itself has a high recognition rate. Using the palm as an auxiliary judgment condition can easily reduce the misjudgment rate.

[0061] Trajectory tracking module, the specific steps are as follows:

[0062] 1. Target category and position determination. Determine whether the category class returned by the recognition module contains the palm class_A. If not, it is considered that there is no target in this frame and waits for the next frame detection. If it contains class_A, save the data of box_A and score_A, and continue to check class_B and class_C. If it exists, save the relevant class, box, score and other related data.

[0063] 2. Combination of targets. Since it is a multi-target detection, there may be multiple class_A, class_B, and class_C. You can use the IOU of different class_A and class_C targets to find the target. A_C To determine which targets are integrated, and thus eliminate the wrong targets. The formula is as follows:

[0064]

[0065] Multiple IOUs A_C The composition set E(IOU A_C ), set E(IOU A_C ) Internally press IOU A_C Sort by values ​​in ascending order.

[0066] 3. Deduplication. The set E(class_B) of multiple class_Bs all represent palm-handheld objects, so there may be duplication. By suppressing nms with non-maximum value, duplicate targets in both sets are removed. nms is generally used to remove overlapping boxes through score.

[0067] There is a score score_B for category B in E(class_B), but there is

[0068] IOU A_C Cannot compare with score_B, can set Compare again.

[0069] The behavior determination module confirms the trajectory in consecutive frames. There are three states of trajectory: take, put, and uncertain. The state is confirmed by trend determination, that is, relying on the current target box. now The center point is compared with the target box in the previous frame pre Assuming that the set of suspected targets in the current frame is E(object) and the set of current tracks is E(track), the specific determination steps are as follows:

[0070] 1. Trend determination, if box now .x>box pre .x and box now .y>box pre .y:take

[0071] if box now .x<box pre .x and box now .y<box pre .y: put

[0072] if(box now .x-box pre .x)*(box now .y-box pre .y)<0:unsure

[0073] 2. Track update: If the movement trend of the first object obj in E(object) is consistent with a track track_x in E(track), then the object is considered to belong to the track and the track is updated. If they are inconsistent, then select the target in E(object) and compare it with the track of E(track). If there is no match, the current E(object) is listed as a suspected track and added to E(track).

[0074] 3. Track culling: When there are multiple consecutive frames in E(track) showing a trend, the behavior is determined and the track is deleted.

[0075] The beneficial effects of the present invention are:

[0076] The device is simple, inexpensive, and has a wide range of applications. It can be deployed on cabinets where food is stored. It is calculated through a dedicated cloud server with powerful computing power and can identify different types of items with good recognition effect. After recognition, it can store food type, cabinet ID, food characteristics, placement time, and removal time data in the storage server, thereby realizing further intelligence of the home.

[0077] The above embodiments of the present invention are not intended to limit the protection scope of the present invention, and the implementation modes of the present invention are not limited thereto. All other modifications, replacements or changes made to the above structures of the present invention based on the above contents of the present invention, in accordance with common technical knowledge and customary means in the art, without departing from the above basic technical ideas of the present invention, should fall within the protection scope of the present invention.

Claims

1. A cabinet food identification system, characterized in that: It includes an identification mechanism, a video streaming server, an identification server and a storage server, wherein: The identification mechanism is used to collect image information of food entering and leaving the cabinet and transmit it to the video streaming server; The video stream server receives the image information collected by the recognition mechanism and distributes the image information to the recognition server; The recognition server receives the image information, recognizes the picture according to the image information, performs tracking judgment and recognition judgment through the picture, and finally sends the judgment result to the storage server; The storage server receives the judgment result and updates the food data information in the cabinet; The video processing system is as follows: The entire system is divided into imaging module, labeling module, training module, recognition module, trajectory tracking module, and behavior determination module; The imaging module is placed in a fixed position through an external network camera to shoot and capture moving targets within the field of view; The labeling module provides training data to the training module. It is divided into three parts. The first part is to label the palm, the second part is to label the hand-held object, and the third part is to label different items in different categories. The training module selects different models for training the labeled images; The training categories include three categories: palm, palm-held objects, and objects; the first two categories are fixed, and the last category of objects is selected according to actual needs; The recognition module uses the trained model to recognize the images. The objects include palm A, palm-held object B, and held object C. It gives the recognized target location box, category class, and score score. Trajectory tracking module, the specific steps are as follows: (1) Target category and position determination: Determine whether the category class returned by the recognition module contains the palm class_A. If not, it is considered that there is no target in this frame and wait for the next frame detection. If it contains class_A, save the data of box_A and score_A, and continue to check class_B and class_C. If it exists, save the relevant class, box, score and other related data. (2) Combination of targets: by using different IOUs of class_A and class_C targets A_C To determine which targets are integrated, so as to eliminate the wrong targets, the formula is as follows: , Multiple IOUs A_C The composition set E(IOU A_C ), set E(IOU A_C ) Internally press IOU A_C Sort by value in ascending order; (3) Deduplication, due to the collection The set E(class_B) of multiple class_Bs represents palm-handheld objects, so there may be duplication. The non-maximum suppression of nms is used to remove the repeated targets in both sets. nms is used to remove overlapping boxes through score. There is a score score_B of category B in the set E(class_B), but there is IOU in the set E(class_B). A_C Unable to compare with score_B, set Compare again; The behavior determination module confirms the trajectory in the continuous frames. The trajectory has three states: take, put, and uncertain. The state is confirmed by trend determination, that is, relying on the current target. The center point is compared with the target in the previous frame The location is poor.

2. A cabinet food identification system as claimed in claim 1, characterized in that: The cabinet is a cabinet, a refrigerator, a food cabinet, and a storage cabinet.

3. A cabinet food identification system as claimed in claim 2, characterized in that: The identification mechanism, video stream server, identification server and storage server are all remote cloud servers, and data is transmitted between each other through internal and external means.

4. A cabinet food identification system as claimed in claim 3, characterized in that: The identification mechanism comprises at least one camera which is arranged outside the box and located on the top of the box.

5. A cabinet food identification system according to any one of claims 1 to 4, characterized in that: The identification mechanism comprises at least one camera which is arranged in the box and located at the bottom of the box.

6. A cabinet food identification system as claimed in claim 5, characterized in that: The food data information includes type, cabinet ID, food characteristics, time of putting in and time of taking out.

7. A method for identifying food in a cabinet, characterized in that: The application of the cabinet food identification system as claimed in claim 1 comprises the following steps: 1) To acquire images, aim the camera at the area in front of the cabinet door to capture the scene of items being put into or taken out of the cabinet. There should be at least one camera, and each camera has an ID, which is bound to the cabinet and connected to the remote server; 2) Transmit and distribute images. The images obtained by each cabinet need to be uploaded to the video streaming server, which will distribute them to the recognition server. 3) Identify the food. The recognition model loaded by the recognition server identifies the corresponding picture. If the target is found, the ID and picture information of the current cabinet are retained and the food tracking mode is turned on. 4) Tracking food, judging the state of food by the movement trend of food in multiple consecutive frames. When the model recognizes that the cabinet door is open, it is judged that food is placed when the food is close to the cabinet, and that food is taken away from the cabinet; 5) Manage food, identify the server, and after obtaining the food status, give feedback according to different statuses. When it is judged that food is to be placed, the relevant data of the food is sent to the storage server, the cabinet food information is stored and updated, and the result is sent to the terminal for display at the same time; when it is judged that food is to be taken, the relevant data of the food is sent to the storage server, and the characteristics of the same type of food returned by the storage server are received. The most likely food is selected by comparing the same type of food, and returned to the storage server, the food information is stored and updated, and the result is sent to the terminal for display.

8. A method for identifying food in a cabinet as claimed in claim 7, characterized in that: In step 4), the food tracking method is: The direction of food in and out is determined by the location of the camera. Let P(x,y) represent the coordinates of the food in the camera. Indicates the food coordinates of the previous frame, Indicates the food coordinates of the current frame, It represents the difference between the two. The specific formula is as follows: If the camera thinks that the food is placed near the camera, If the value is positive, it means that the person is taking food when he is far away from the camera. The value is negative; when the coordinates of the food in N consecutive frames are all positive, N is the algorithm preset value, then it is considered that the current state is to place food. When the food coordinates in N consecutive frames are If it is negative, it is considered that the current state is to take food.

9. A cabinet food identification method as claimed in claim 7, characterized in that: In step 5), the terminal includes a mobile phone APP, a WeChat applet, and a display screen of the cabinet.