Food material putting identification method, device and system and storage medium

Abstract

The invention discloses a food material putting recognition method, device and system and a storage medium, and belongs to the technical field of household appliances, and the method comprises the steps: obtaining multiple frames of first images, carrying out the hand and food material detection recognition of the multiple frames of first images, and obtaining the hand and food material detection recognition results corresponding to the first images in the multiple frames of first images; according to hand and food material detection and recognition results corresponding to the first images in the multiple frames of first images, whether the action of the user is food material putting or not is determined, and the food material putting position comprises a rack edge or a non-rack edge; and under the condition that the action of the user is determined to be food material putting, determining a target shelf area where the food material is put based on a target image of a bounding box of the hand-held food material in the multi-frame first image and a food material detection and recognition result. By detecting the hand and the food materials, the action of the hand can be recognized more accurately, so that whether the food materials are put in or not is judged, and the recognition accuracy is improved.

Description

Technical Field

[0001] This application belongs to the field of household appliance technology, and in particular relates to a method, device, system and storage medium for identifying food ingredients placed in the container. Background Technology

[0002] In related technologies, refrigerator food retrieval and placement recognition generally uses image comparison technology for static recognition. Each shelf needs to be equipped with a camera. These food retrieval and placement recognition methods have problems such as high data collection costs, food that is close to the camera may be obstructed from the view and cannot be captured, and inaccurate recognition results, which bring many inconveniences to the practical application of refrigerator food recognition. Summary of the Invention

[0003] This application proposes a method, device, system, and storage medium for identifying food items placed in refrigerators, in order to solve the problems of high acquisition costs, inaccurate identification results, and potential obstruction of the field of view by food items near the camera.

[0004] Firstly, this application provides a method for identifying food ingredients placed in the container, the method comprising:

[0005] Acquire the first image of multiple frames;

[0006] The hands and food ingredients are detected and identified in the multiple frames of the first image, and the detection and identification results of the hands and food ingredients in each first image are obtained.

[0007] Based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first image, it is determined whether the user's action is to put food in, and the position where the food is put in includes the edge of the shelf or a non-shelf edge.

[0008] If the user's action is determined to be placing food, the target shelf area for placing the food is determined based on the target image of the bounding box of the hand holding the food in the multi-frame first image, including the hand and food detection and recognition results.

[0009] The food placement recognition method provided in this application acquires multiple frames of first images, detects and recognizes hands and food in each frame, and determines whether the user's action is placing food based on the detection and recognition results. If the user is placing food, the method determines the shelf area where the food is placed. Classifying and detecting hands and food allows for more accurate food placement recognition. Determining whether the user is placing food before determining the placement area avoids unnecessary judgment processes and improves recognition accuracy.

[0010] According to one embodiment of this application, the hand and food ingredient detection and identification results include one of the following:

[0011] The bounding box of holding ingredients;

[0012] The bounding box of the hand;

[0013] The bounding box of the ingredients;

[0014] null.

[0015] In the above technical solution, the electronic device classifies the recognition results of the first image into labels such as hand, food, hand holding food, and empty, which can more clearly reflect the recognition results and facilitate subsequent recognition and judgment. It can realize the recognition of food placed on the edge of the shelf.

[0016] According to one embodiment of this application, determining whether the user's action is to put food in the food based on the hand and food detection and recognition results corresponding to each of the multiple first images includes:

[0017] If the hand and food detection and recognition results corresponding to each first image in the multi-frame first image meet the first preset condition, then it is determined that the user's action is not putting in food.

[0018] The first preset conditions include:

[0019] In the hand and food detection and recognition results corresponding to each of the multiple first images, if there are N consecutive frames of hand and food detection and recognition results, the results will be one of the following:

[0020] Empty or food bounding box, hand bounding box, empty;

[0021] The bounding box of the empty space or food, the bounding box of the hand, the bounding box of the hand;

[0022] The bounding box of an empty or food item, the bounding box of the first hand holding the food item, the bounding box of the second hand holding the food item, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food item and the Y value of the bounding box of the second hand holding the food item is greater than or equal to a first value.

[0023] In the above technical solution, by setting a first preset condition to determine whether the user's action is not putting in food, when there are N consecutive frames of hand and food detection and recognition results in the first images of multiple first images that meet the first preset condition, it is determined that the user's action is not putting in food, which can improve the recognition accuracy.

[0024] According to one embodiment of this application, determining whether the user's action is to put food in the food based on the hand and food detection and recognition results corresponding to each of the multiple first images includes:

[0025] If the hand and food detection and recognition results corresponding to each first image in the multi-frame first image meet the second preset condition, then it is determined that the user's action is to put in food.

[0026] The second preset condition includes:

[0027] Among the hand and food detection and recognition results corresponding to the multiple first images, the hand and food detection and recognition results of N consecutive images are one of the following:

[0028] Empty or bounding box of ingredients, bounding box of hand holding ingredients, empty;

[0029] The bounding box of an empty or food item, the bounding box of a hand holding a food item, and the bounding box of the food item.

[0030] The bounding box of an empty space or food ingredient, the bounding box of a hand holding food ingredient, and the bounding box of a hand.

[0031] The bounding box of an empty space or food item, the bounding box of a hand holding food item, the bounding box of a hand, and the bounding box of a food item.

[0032] The bounding box of an empty or food item, the bounding box of the first hand holding the food item, the bounding box of the second hand holding the food item, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food item and the Y value of the bounding box of the second hand holding the food item is less than or less than or equal to a first value.

[0033] In the above technical solution, by setting a second preset condition to determine whether the user's action is to put food in, when there are N consecutive frames of hand and food detection and recognition results in the first images of multiple first images that meet the second preset condition, it is determined that the user's action is to put food in, and then other detections are performed, which can improve the recognition accuracy and increase the detection efficiency.

[0034] According to one embodiment of this application, the step of determining the target shelf area where the food is placed based on the hand and food detection and recognition results in the multiple frames of the first image, including the target image of the bounding box of the hand holding the food, includes:

[0035] An image segmentation algorithm is used to segment the shelf area in the target image to obtain the shelf area segmentation result corresponding to the target image;

[0036] Based on the Y value of the bounding box of the hand holding the food in the target image and the segmentation result of the shelf area corresponding to the target image, the target shelf area where the food is placed is determined.

[0037] The target image is the last frame of the first image of the multi-frames before the ID of the same hand disappears, and the image of the bounding box of the hand holding the food is the result of the hand and food detection and recognition.

[0038] In the above technical solution, the shelf area in the target image is segmented by an image segmentation algorithm, which makes it easier to determine the specific location where the food is placed. By using the Y value of the bounding box of the hand holding the food in the target image and the segmentation result of the corresponding shelf area in the target image, the target shelf area where the food is placed is determined, which reduces errors and improves recognition accuracy. The image of the bounding box of the hand holding the food in the last frame before the ID of the same hand disappears in the first image of multiple frames is used as the target image. On the basis of accurate detection results, the computation of electronic equipment can be further simplified and the recognition efficiency can be improved.

[0039] According to one embodiment of this application, the first image is an image captured by a first camera, or obtained by processing the video captured by the first camera. The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf. The first camera is used to capture images of the shelf area after the door of the refrigeration unit is opened.

[0040] In the above technical solution, by using a first camera installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf to capture the first image, the field of view of the refrigerator shelf area can be accurately obtained, avoiding missed detection.

[0041] According to one embodiment of this application, the method further includes:

[0042] Control the feedback component corresponding to the target shelving area to output feedback information.

[0043] In the above technical solution, the user experience can be improved and the practicality can be enhanced by controlling the feedback component corresponding to the target shelving area to output feedback information.

[0044] Secondly, this application provides a food ingredient placement identification device, the device comprising:

[0045] An image acquisition unit is used to acquire multiple frames of the first image;

[0046] The detection and recognition unit is used to detect and recognize hands and food in the multiple frames of first images respectively, and to obtain the detection and recognition results of hands and food in each first image in the multiple frames of first images;

[0047] The first determining unit is used to determine whether the user's action is to put food into the shelf based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first image. The location where the food is put into the shelf includes the edge of the shelf or a location other than the edge of the shelf.

[0048] The second determining unit is used to determine the target shelf area where the food is placed, based on the target image of the bounding box of the hand holding the food in the multi-frame first image, when the user's action is determined to be putting food in.

[0049] In the above technical solution, the first image of the refrigeration device is acquired by the image acquisition unit, and the hand and food are detected and identified by the detection and recognition unit on multiple frames of the first image. The first determination unit determines whether the user's action is to put in food, and the second determination unit determines the shelf area where the food is to be put in. Based on the hand trajectory, it is first determined whether the hand action is to put in or take out food. If it is identified that food is to be put in, the shelf area where the user put in is further determined. This can improve the recognition accuracy, avoid false detection, and reduce the processing volume.

[0050] Thirdly, this application provides a food placement recognition system, including: a refrigeration device, a first camera, and electronic equipment, wherein...

[0051] The first camera is installed at the top center of the inner liner of the refrigeration unit and tilted towards the shelf. The first camera is used to photograph the shelf area after the door of the refrigeration unit is opened.

[0052] The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the food placement recognition method as described in the first aspect.

[0053] In the above technical solution, the first camera captures the shelf area after the door of the refrigeration unit is opened, and the electronic device identifies the food placed into the refrigeration unit by processing the image captured by the first camera. The recognition accuracy can be improved by recognizing the hand and the food.

[0054] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the food ingredient placement recognition method as described in the first aspect above.

[0055] Fifthly, this application provides a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the food placement recognition method as described in the first aspect.

[0056] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the food ingredient placement recognition method as described in the first aspect above.

[0057] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0058] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0059] Figure 1 This is a flowchart of the food ingredient placement and recognition method provided in the embodiments of this application;

[0060] Figure 2 This is a schematic diagram illustrating the division of the shelving area according to an embodiment of this application;

[0061] Figure 3 This is a schematic diagram illustrating the area of ​​the shelf where food items are placed, provided in an embodiment of this application.

[0062] Figure 4 This is a schematic diagram of the installation position of the first camera provided in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram of the shooting field of view of the first camera provided in an embodiment of this application;

[0064] Figure 6 This is a schematic diagram of a food ingredient placement recognition device provided in an embodiment of this application;

[0065] Figure 7 This is a schematic diagram of a food ingredient placement recognition system provided in an embodiment of this application;

[0066] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application.

[0067] Explanation of reference numerals in the attached figures:

[0068] 201: LED strip; 60: Food ingredient placement recognition device; 601: Image acquisition unit;

[0069] 602: Detection and recognition unit; 603: First determination unit; 604: Second determination unit.

[0070] 70: Food ingredients are placed into the recognition system; 701: Refrigeration unit; 702: First camera;

[0071] 703: Electronic device; 801: Processor; 802: Memory. Detailed Implementation

[0072] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0073] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0074] The following description, in conjunction with the accompanying drawings, details the food placement identification method, food placement identification device, and food placement identification system provided in this application through specific embodiments and application scenarios.

[0075] The food placement identification method can be applied to a terminal, specifically executed by the hardware or software within the terminal.

[0076] The food placement identification method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the food placement identification method. The electronic devices mentioned in this application embodiment include, but are not limited to, controllers in refrigeration devices, processors in refrigeration devices, edge servers, backend servers, cloud servers, mobile phones, tablets, computers, cameras, and wearable devices. The food placement identification method provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0077] Figure 1 This is a flowchart of the food ingredient placement and recognition method provided in the embodiments of this application, such as... Figure 1 As shown, the food ingredient identification method includes steps 110, 120, 130, and 140.

[0078] Step 110: Obtain multiple frames of the first image;

[0079] In this embodiment of the application, the electronic device acquires multiple frames of the first image of the cooling device. The first image may be an image acquired by a acquisition device installed on the cooling device, or an image obtained by processing the video acquired by the acquisition device through the electronic device.

[0080] The data acquisition device can be a camera or a sensor capable of generating images or videos. It should be understood that, in order to identify food items placed in the refrigeration unit, the data acquisition device's field of view includes the shelf area after the refrigeration unit's door is opened.

[0081] It should be noted that the refrigeration device in the embodiments of this application includes refrigerators, freezers, and other devices that can refrigerate and have a cabinet and door. The refrigeration device in the embodiments of this application can also be replaced by a cargo container, all of which are within the protection scope of this application.

[0082] The following text uses a refrigerator as an example of a refrigeration device.

[0083] Optionally, when the refrigeration unit is a refrigerator, the shelves inside the refrigerator can be transparent. If the shelves inside the refrigerator are transparent, the data acquisition device can capture the field of view inside the refrigerator shelves.

[0084] Optionally, the shelves inside the refrigerator can also be non-transparent, which can reduce interference with target detection and recognition in the field of view. In this embodiment, the shelves of the refrigerator are set to be non-transparent. When food is put into the shelves, the food will be blocked, and the hand will also be blocked. When the food is placed on the edge of the shelf, the collection device can capture the food.

[0085] It is worth noting that the electronic device can be any device capable of meeting the needs of data transmission and data processing (such as data calculation, deletion or storage, product identification and positioning). For example, it can be the refrigerator's controller / processor, or a functional module or entity of the refrigerator's controller / processor, or a terminal device, server or other hardware device capable of communicating with the refrigerator's controller / processor and data acquisition device.

[0086] Step 120: Detect and identify hands and food ingredients in the multiple frames of the first image respectively, and obtain the detection and identification results of hands and food ingredients in each first image in the multiple frames of the first image;

[0087] It should be noted that when the electronic device detects and recognizes multiple frames of the first image, it will detect and recognize the hand and food in the multiple frames of the first image separately. The recognition result of the electronic device includes one of the following: hand holding food, hand, food, or one in the air.

[0088] Optionally, when a hand holding food appears in the first image, the electronic device will mark the hand holding food with a rectangle; when a hand appears in the first image, the electronic device will mark the hand with a rectangle; when food appears in the first image, for example, when food is placed on the edge of a refrigerator shelf, the electronic device will mark the food with a rectangle; when there is neither a hand nor food in the first image, the electronic device will not make a mark, and the recognition result will be empty.

[0089] In this step, taking a refrigerator as an example, the refrigerator shelf area can be divided. The electronic device can comprehensively determine which shelf the user has placed the food on based on the detection and recognition results of the hand and food in multiple frames of the first image.

[0090] Step 130: Based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first image, determine whether the user's action is to put food in, and the position where the food is put in includes the edge of the shelf or a non-shelf edge.

[0091] It is worth noting that when a hand or a hand plus food is detected in the first image, it indicates that the user may have placed food inside. To determine whether the hand action is for placing food or not, the electronic device will judge whether the hand and food detection and recognition results in each of the multiple frames of the first image meet the preset conditions for placing food.

[0092] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to the first image of multiple frames, there are three consecutive images in which the hand and food detection and recognition results are empty, the bounding box of the hand holding the food, and empty, at which point the user has performed the action of putting the food in and placing the food in the shelf.

[0093] Optionally, the preset condition can be that the hand and food detection and recognition results of the first image in multiple frames contain three consecutive frames of images where the hand and food detection and recognition results are, in order, the bounding box of the food, the bounding box of the hand holding the food, and an empty frame. At this time, the user has performed the action of putting the food in and placing the food in the shelf. The bounding box of the food is detected before the bounding box of the hand holding the food because there is already food on the edge of the refrigerator shelf.

[0094] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to the first image of multiple frames, there are three consecutive images in which the hand and food detection and recognition results are empty, the bounding box of the hand holding the food, and the bounding box of the food. At this time, the user has performed the action of putting the food in and placing the food on the edge of the shelf.

[0095] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple first images, there are three consecutive images where the hand and food detection and recognition results are, in order, the bounding box of the food, the bounding box of the hand holding the food, and the bounding box of the food. At this point, the user has performed the action of placing the food and put it on the edge of the shelf. The bounding box of the food is detected before the bounding box of the hand holding the food because there is already food on the edge of the refrigerator shelf.

[0096] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple first images, there are three consecutive images where the hand and food detection and recognition results are empty, the bounding box of the hand holding the food, and the bounding box of the hand. At this time, the user has put the food into the shelf. After the food is put in, the hand has not completely left the collection field of view of the acquisition device. Therefore, the bounding box of the hand exists in the first image.

[0097] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple first images, there are three consecutive images where the hand and food detection and recognition results are, in order, the bounding box of the food, the bounding box of the hand holding the food, and the bounding box of the hand. At this time, the user has performed the action of putting the food into the shelf. After the food is put in, the hand has not completely left the collection field of view of the acquisition device. Therefore, the bounding box of the hand exists in the first image. The bounding box of the food is detected before the bounding box of the hand holding the food because there is already food on the edge of the refrigerator shelf.

[0098] Optionally, the preset condition can be that the hand and food detection and recognition results of three consecutive frames in the first image are empty, the bounding box of the hand holding the food, the bounding box of the hand, and the bounding box of the food. At this time, the user puts the food in and places it on the edge of the shelf. Therefore, after the food is put in, the food is present in the first image. At this time, the hand has not completely left the acquisition field of the acquisition device, and the bounding box of the food and the bounding box of the hand can be detected.

[0099] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple first images, there are three consecutive images where the hand and food detection and recognition results are, in order, the bounding box of the food, the bounding box of the hand holding the food, the bounding box of the hand, and the bounding box of the food. At this time, the user performs the action of placing the food and puts it on the edge of the shelf. Therefore, after the food is placed, it is present in the first image. At this time, the hand has not completely left the field of view of the acquisition device, and the bounding boxes of the food and the hand can be detected. The bounding box of the food is detected before the bounding box of the hand holding the food because there is already food on the edge of the refrigerator shelf.

[0100] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to the first image of multiple frames, there are three consecutive images in which the hand and food detection and recognition results are empty, the bounding box of the first hand holding the food, and the bounding box of the second hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food and the Y value of the bounding box of the second hand holding the food is less than or less than or equal to the first value. At this time, the user has performed the action of putting the food in, but has not put the food in completely.

[0101] It should be noted that the Y value of the bounding box can be a coordinate value of the bounding box. If the bounding box is given in the form of center point coordinates (x_center, y_center) and width and height, then the Y value of the bounding box represents the vertical coordinate of the center point of the bounding box in the image.

[0102] Optionally, if the bounding box is given in the form of top-left corner coordinates (x1, y1) and bottom-right corner coordinates (x2, y2), then the Y value of the bounding box refers to the vertical coordinate of the top-left corner of the bounding box.

[0103] It is worth noting that the first value can be a number set according to the actual situation.

[0104] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple first images, there are three consecutive images where the hand and food detection and recognition results are, in order, the bounding box of the food, the bounding box of the first hand holding the food, and the bounding box of the second hand holding the food. Furthermore, the absolute value of the difference between the Y-value of the bounding box of the first hand holding the food and the Y-value of the bounding box of the second hand holding the food is less than or equal to a first value. In this case, the user has performed the action of placing food, but has not completely placed the food. Detecting the food's bounding box before the bounding box of the hand holding the food indicates that food is already present on the edge of the refrigerator shelf.

[0105] It should be noted that this embodiment only identifies when the user places food into the container. The process ends when the user's action is not placing food. The electronic device will determine whether the hand and food detection and recognition results corresponding to each of the multiple first images meet the preset conditions for not placing food.

[0106] Optionally, the preset condition can be that the hand and food detection and recognition results of three consecutive frames of the first image are empty, the bounding box of the hand, and empty in sequence. At this time, the user's action may be taking it out of the refrigerator or putting his hand into the refrigerator to take out food.

[0107] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple first images, there are three consecutive images where the hand and food detection and recognition results are, in order, the bounding box of the food, the bounding box of the hand, and an empty space. At this time, the user's action may be taking something out of the refrigerator or putting their hand into the refrigerator to retrieve food. Food is detected before the bounding box of the hand because food is already present on the edge of the refrigerator shelf, or because the hand is taken out of the refrigerator after the food is placed on the edge of the refrigerator shelf.

[0108] Optionally, the preset condition can be that the hand and food detection and recognition results of three consecutive frames of the first image are empty, the bounding box of the hand, and the bounding box of the hand in sequence. At this time, the user's action may be taking the hand out of the refrigerator or putting the hand into the refrigerator to take out food.

[0109] Optionally, the preset condition can be that the hand and food detection and recognition results of three consecutive frames of the first image are the bounding boxes of the food, the bounding boxes of the hand, and the bounding boxes of the hand in sequence. At this time, the user's action may be taking the hand out of the refrigerator or putting the hand into the refrigerator to take out food.

[0110] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple first images, there are three consecutive images in which the hand and food detection and recognition results are empty, the bounding box of the first hand holding food, and the bounding box of the second hand holding food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding food and the Y value of the bounding box of the second hand holding food is greater than or equal to the first value. In this case, the user's action may be taking food out of the refrigerator.

[0111] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple first images, there are three consecutive images in which the hand and food detection and recognition results are, in order, the bounding box of the food, the bounding box of the first hand holding the food, and the bounding box of the second hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food and the Y value of the bounding box of the second hand holding the food is greater than or equal to the first value. In this case, the user's action may be taking food out of the refrigerator.

[0112] Step 140: If the user's action is to put food in, based on the target image of the hand and food detection and recognition results in the multi-frame first image, including the bounding box of the hand holding the food, determine the target shelf area where the food is to be placed.

[0113] In this step, once it is determined that the user's action is to place the food, the electronic device will further determine the specific location where the food is placed.

[0114] Optionally, the electronic device uses an image segmentation algorithm to divide the shelf area and adds an index.

[0115] Optionally, the target image is the last frame of the first multi-frame image before the same hand disappears, showing the bounding box of the hand holding the food. During the user's food placement process, the electronic device detects multiple frames of the bounding box of the hand holding the food, with the last frame of the bounding box before the same hand disappears being closest to the final position of the food placement.

[0116] In this embodiment of the application, in the last frame of the first image before the same hand disappears, the image of the bounding box of the hand holding the food can be identified by judging the correspondence between the Y value of the bounding box and the shelf area, and thus the final shelf area where the food is placed can be determined.

[0117] Optionally, light strips can be installed along the edge of each shelf. When the electronic device detects that food has been placed on a shelf, it controls the light strip on that shelf to light up to remind the user.

[0118] The food placement recognition method provided in this application acquires multiple frames of first images through an image or video acquisition device. Hand and food items are detected and recognized in each of the multiple first frames. Based on the detection and recognition results, it is determined whether the user's action is placing food. If the user is placing food, the shelf area where the food is placed is determined. The classification and detection of hands and food items enables more accurate identification of food placement. Determining whether the user is placing food before determining the placement area avoids unnecessary judgment processes and improves recognition accuracy.

[0119] According to one embodiment of this application, the hand and food ingredient detection and identification results include one of the following:

[0120] The bounding box of holding ingredients;

[0121] The bounding box of the hand;

[0122] The bounding box of the ingredients;

[0123] null.

[0124] It is worth noting that when the electronic device detects the first image captured by the first camera, it will classify and identify the food as being held by hand, hand, food, or empty space.

[0125] Optionally, when a hand appears in the first image, the electronic device marks the hand with a rectangle and assigns an ID to the hand.

[0126] Optionally, when food appears in the first image, the electronic device marks the food with a rectangle and identifies its category. It should be noted that when food appears in the first image, it is typically located on the edge of a refrigerator shelf. This embodiment of the application, by detecting the food, can identify and place food placed on the edge of a shelf.

[0127] Optionally, when a hand holding food appears in the first image, the electronic device marks the hand holding food with a rectangle.

[0128] It's important to note that electronic devices can determine whether food is being handled by hand based on the degree of overlap between the hand and the food. For example, they can calculate the intersection-union ratio (IUR) between the rectangles representing the hand and the food. The IUR is calculated by dividing the area of ​​their intersection by the area of ​​their union; the value ranges from 0 to 1, with a higher value indicating greater overlap. The electronic device determines whether food is being handled by hand by comparing the IUR to a preset threshold.

[0129] Optionally, the preset threshold can be set between 0 and 1 according to the actual situation. For example, if the preset threshold is set to 0.5, the intersection-union ratio between the rectangle of the hand and the rectangle of the food in the first image is 0.6, which is greater than the preset threshold. At this time, it is determined that the hand is holding the food, and the rectangle is used to mark the hand holding the food.

[0130] In one embodiment of this application, the rectangle of the hand used to calculate whether it is holding food can also be a triangle. When the electronic device detects that the hand and food overlap in the first image, it can mark the food with a rectangle. The device determines whether the food is being held by hand by comparing the intersection-union ratio (IU) between the rectangle of the food and the triangle of the hand with a preset threshold. For example, if the IU is 0.75, which is greater than the preset threshold of 0.5, the electronic device determines that the food is being held by hand and marks the food with a rectangle.

[0131] Optionally, when there is neither a hand nor food in the first image, the electronic device will recognize the image as empty and will not label the first image.

[0132] In the above technical solution, the electronic device classifies the recognition results of the first image into labels such as hand, food, hand holding food, and empty, which can more clearly reflect the recognition results and facilitate subsequent recognition and judgment. It can realize the recognition of food placed on the edge of the shelf.

[0133] In one embodiment of this application, determining whether the user's action is to put food in the food based on the hand and food detection and recognition results corresponding to each of the multiple first images includes:

[0134] If the hand and food detection and recognition results corresponding to each first image in the multi-frame first image meet the first preset condition, then it is determined that the user's action is not putting in food.

[0135] The first preset conditions include:

[0136] In the hand and food detection and recognition results corresponding to each of the multiple first images, if there are N consecutive frames of hand and food detection and recognition results, the results will be one of the following:

[0137] Empty or food bounding box, hand bounding box, empty;

[0138] The bounding box of the empty space or food, the bounding box of the hand, the bounding box of the hand;

[0139] The bounding box of an empty or food item, the bounding box of the first hand holding the food item, the bounding box of the second hand holding the food item, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food item and the Y value of the bounding box of the second hand holding the food item is greater than or equal to a first value.

[0140] It is worth noting that when the electronic device determines whether the user's action is to put food in or not, it needs to make a judgment through multiple consecutive first images. If the hand and food detection and recognition results of each first image in the multiple first images meet the first preset condition, then the user's action is determined to be not to put food in.

[0141] Optionally, the first preset condition is the set of all possibilities where the user's action is not putting in food. The first preset condition includes the existence of N consecutive frames of hand and food detection and recognition results in the first images corresponding to each first image in multiple frames, where the hand and food detection and recognition results are specific results in sequence.

[0142] Where N is a positive integer greater than 1.

[0143] The ingredients not included include at least one of the following:

[0144] After placing the ingredients in the shelf area, leave the shelf area empty-handed.

[0145] Enter the shelving area empty-handed, ready to retrieve ingredients;

[0146] Take the ingredients and leave the shelf area.

[0147] Optionally, when a user places food in the shelf area and leaves empty-handed, or when a user enters the shelf area empty-handed and prepares to retrieve food, the first preset condition can be that in the hand and food detection and recognition results corresponding to each of the first images in multiple first images, there are consecutive N frames of hand and food detection and recognition results where the hand and food detection and recognition results are empty, and the bounding box of the hand is empty.

[0148] Optionally, when a user places food on the edge of the shelf and leaves the shelf area empty-handed, or when a user enters the shelf area empty-handed and prepares to take food, the first preset condition can be that the hand and food detection and recognition results of each of the first images in multiple first images contain N consecutive images whose hand and food detection and recognition results are, in order, the bounding box of the food, the bounding box of the hand, and empty.

[0149] Optionally, when a user places food in the shelf area and leaves empty-handed, or when a user enters the shelf area empty-handed and prepares to retrieve food, the first preset condition may be that in the hand and food detection and recognition results corresponding to each of the first images in multiple first images, there are consecutive N frames of hand and food detection and recognition results where the results are empty, the bounding box of the hand, and the bounding box of the hand.

[0150] Optionally, when a user places food on the edge of the shelf and leaves the shelf area empty-handed, or when a user enters the shelf area empty-handed and prepares to take food, the first preset condition can be that the hand and food detection and recognition results of each of the first images in multiple first images contain N consecutive images whose hand and food detection and recognition results are, in order, the bounding box of the food, the bounding box of the hand, and the bounding box of the hand.

[0151] Optionally, when a user leaves the shelf area with food in their hand, the first preset condition may be that in the hand and food detection and recognition results corresponding to each of the first images in multiple first images, there are consecutive N frames of hand and food detection and recognition results that are empty in sequence, the bounding box of the first hand holding food, the bounding box of the second hand holding food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding food and the Y value of the bounding box of the second hand holding food is greater than or equal to the first value.

[0152] It should be noted that the Y value of the bounding box can be a coordinate value of the bounding box. If the bounding box is given in the form of center point coordinates (x_center, y_center) and width and height, then the Y value of the bounding box represents the vertical coordinate of the center point of the bounding box in the image.

[0153] Optionally, if the bounding box is given in the form of top-left corner coordinates (x1, y1) and bottom-right corner coordinates (x2, y2), then the Y value of the bounding box refers to the vertical coordinate of the top-left corner of the bounding box.

[0154] It is worth noting that the first value can be a number set according to the actual situation.

[0155] In the above technical solution, by setting a first preset condition to determine whether the user's action is not putting in food, when there are N consecutive frames of hand and food detection and recognition results in the first images of multiple first images that meet the first preset condition, it is determined that the user's action is not putting in food, which can improve the recognition accuracy.

[0156] In one embodiment of this application, determining whether the user's action is to put food in the food based on the hand and food detection and recognition results corresponding to each of the multiple first images includes:

[0157] If the hand and food detection and recognition results corresponding to each first image in the multi-frame first image meet the second preset condition, then it is determined that the user's action is to put in food.

[0158] The second preset condition includes:

[0159] Among the hand and food detection and recognition results corresponding to the multiple first images, the hand and food detection and recognition results of N consecutive images are one of the following:

[0160] Empty or bounding box of ingredients, bounding box of hand holding ingredients, empty;

[0161] The bounding box of an empty or food item, the bounding box of a hand holding a food item, and the bounding box of the food item.

[0162] The bounding box of an empty space or food ingredient, the bounding box of a hand holding food ingredient, and the bounding box of a hand.

[0163] The bounding box of an empty space or food item, the bounding box of a hand holding food item, the bounding box of a hand, and the bounding box of a food item.

[0164] The bounding box of an empty or food item, the bounding box of the first hand holding the food item, the bounding box of the second hand holding the food item, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food item and the Y value of the bounding box of the second hand holding the food item is less than or less than or equal to a first value.

[0165] It is worth noting that when the electronic device determines whether the user's action is to put food in or not, it needs to make a judgment through multiple consecutive first images. If the hand and food detection and recognition results of each first image in the multiple first images meet the second preset condition, then the user's action is determined to be putting food in.

[0166] Optionally, the second preset condition is that the user's action is a set of various possibilities for putting in ingredients. The second preset condition includes that the hand and ingredient detection and recognition results of N consecutive frames of images in the first images of multiple frames are specific results.

[0167] Where N is a positive integer greater than 1.

[0168] Optionally, when the user puts in the food, the second preset condition can be that in the hand and food detection and recognition results of each of the first images in multiple frames of the first image, there are consecutive N frames of hand and food detection and recognition results that are empty in sequence, and the bounding box of the hand holding the food is empty.

[0169] Optionally, when the user puts in the food, the second preset condition can be that the hand and food detection and recognition results of each of the first images in the multi-frame first images are N consecutive frames of hand and food detection and recognition results, which are in the order of food bounding box, hand holding food bounding box, and empty.

[0170] Optionally, when the user puts in the food, the second preset condition can be that in the hand and food detection and recognition results corresponding to each of the first images in multiple first images, there are consecutive N frames of hand and food detection and recognition results that are empty in sequence, the bounding box of the hand holding the food, and the bounding box of the food.

[0171] Optionally, when the user puts in the food, the second preset condition can be that the hand and food detection and recognition results of each of the first images in the multi-frame first images are N consecutive frames of hand and food detection and recognition results, which are in the order of food bounding box, hand holding food bounding box, and food bounding box.

[0172] Optionally, when the user puts in the food, the second preset condition can be that in the hand and food detection and recognition results corresponding to each of the first images in multiple first images, there are consecutive N frames of hand and food detection and recognition results that are empty in sequence, the bounding box of the hand holding the food, and the bounding box of the hand.

[0173] Optionally, when the user puts in the food, the second preset condition can be that the hand and food detection and recognition results of each of the first images in the multi-frame first images are N consecutive frames of hand and food detection and recognition results, which are in the order of food bounding box, hand holding food bounding box, and hand bounding box.

[0174] Optionally, when the user puts in the food, the second preset condition can be that in the hand and food detection and recognition results corresponding to each of the first images in multiple first images, there are N consecutive images where the hand and food detection and recognition results are either empty or the bounding box of the food, the bounding box of the first hand holding the food, and the bounding box of the second hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food and the Y value of the bounding box of the second hand holding the food is less than or less than or equal to the first value.

[0175] In related technologies, food placement recognition in refrigerators requires users to fully insert their hands into the shelves, with the shelves obscuring both the hands and the food. If users habitually place food on the edge of the shelf, it becomes impossible to determine whether food has been placed inside, leading to poor usability and inaccurate recognition results, causing considerable inconvenience to the practical application of refrigerator food placement recognition. However, in this embodiment, by taking into account the case where the hand and food detection result is the bounding box of the food, it is possible to recognize food placed on the edge of the shelf.

[0176] In the above technical solution, by setting a second preset condition to determine whether the user's action is to put food in, when there are N consecutive frames of hand and food detection and recognition results in the first images of multiple first images that meet the second preset condition, it is determined that the user's action is to put food in, and then other detections are performed, which can improve the recognition accuracy and increase the detection efficiency.

[0177] In one embodiment of this application, the step of determining the target shelf area where the food is placed based on the target image of the bounding box of the hand holding the food in the multi-frame first image includes:

[0178] An image segmentation algorithm is used to segment the shelf area in the target image to obtain the shelf area segmentation result corresponding to the target image;

[0179] Based on the Y value of the bounding box of the hand holding the food in the target image and the segmentation result of the shelf area corresponding to the target image, the target shelf area where the food is placed is determined.

[0180] The target image is the last frame of the first image of the multi-frames before the ID of the same hand disappears, and the image of the bounding box of the hand holding the food is the result of the hand and food detection and recognition.

[0181] It is worth noting that in the above embodiments, when determining whether the user's action is to put food in, the electronic device needs to combine multiple consecutive first images for judgment. When it is determined that the user's action is to put food in, the electronic device can further determine the target shelf area where the food is put in. At this time, it is only necessary to detect the shelf area where the hand holding the food is located in the last frame of the first image before the ID of the same hand disappears and the food detection and recognition result is the bounding box of the hand holding the food. This area is taken as the target shelf area where the food is put in.

[0182] It should be noted that after the target image is determined, the electronic device can use an image segmentation algorithm to segment the shelf area in the target image, obtain the shelf area segmentation result corresponding to the target image, and add an index to each segmented shelf area.

[0183] Figure 2 This is a schematic diagram illustrating the division of the shelving area provided in an embodiment of this application, as shown below. Figure 2 As shown, taking a refrigerator as an example, a first-layer shelf area can be set up in the refrigerator. Figure 2 The area between the first and second shelves of the refrigerator (index number 1, above the red horizontal line) is located in the region between the first and second shelves. Figure 2 The area between the red and green horizontal lines (index number 2) is the area between the second and third shelves of the refrigerator. Figure 2 The index number between the green and yellow horizontal lines is 3.

[0184] Optionally, the electronic device marks the hands holding food in the target image with rectangular boxes. Figure 3 This is a schematic diagram illustrating the purpose of placing food items into the shelf area provided in this application embodiment. Figure 3The green rectangle shown is the bounding box of the hand holding the food. Based on the Y-value of the bounding box of the hand holding the food in the target image (e.g., ...), ... Figure 3 The vertical coordinate of the top right corner of the green rectangle shown is located in the shelf area. The target shelf area for placing the food is determined to be the area between the red and green horizontal lines. Based on the shelf area segmentation results above, the index number of the target shelf area for placing the food is 2.

[0185] In the above technical solution, the shelf area in the target image is segmented by an image segmentation algorithm, which makes it easier to determine the specific location where the food is placed. By using the Y value of the bounding box of the hand holding the food in the target image and the segmentation result of the corresponding shelf area in the target image, the target shelf area where the food is placed is determined, which reduces errors and improves recognition accuracy. The image of the bounding box of the hand holding the food in the last frame before the ID of the same hand disappears in the first image of multiple frames is used as the target image. On the basis of accurate detection results, the computation of electronic equipment can be further simplified and the recognition efficiency can be improved.

[0186] In one embodiment of this application, the acquisition device is a first camera, and the first image is an image acquired by the first camera or obtained by processing the video acquired by the first camera. The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf. The first camera is used to capture images of the shelf area after the door of the refrigeration unit is opened.

[0187] In this step, the field of view of the first camera is the first field of view, which can be the view after the refrigerator door is opened, including the shelf area after the refrigeration unit door is opened. The first video can be a video of the shelf area after the refrigeration unit door is opened, captured by the first camera during the process of putting or taking food into or out of the refrigeration unit.

[0188] It should be noted that after the electronic device acquires the first video, it performs frame-by-frame processing on the first video to obtain multiple frames of the first field of view image. The first image is one frame of the multiple frames of the first field of view image.

[0189] Optionally, the first camera can be a wide-angle camera, an infrared camera, a high-definition camera, a panoramic camera, a wireless camera, a night vision camera, a smart camera, etc. The type of the first camera in this application embodiment is not limited.

[0190] Optionally, the first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf. The first camera can be connected to the electronic device via wired or wireless means. The first camera sends the video or image captured in the first field of view to the electronic device, and the electronic device processes the video to obtain the first image.

[0191] Figure 4 This is a schematic diagram of the installation position of the first camera provided in an embodiment of this application, as shown below. Figure 4 As shown, the refrigeration device can be a refrigerator, and the first camera 702 can be installed in the middle of the top of the inner liner of the refrigerator. The first camera 702 acquires images or videos within the field of view of the shelf area after the refrigerator door is opened.

[0192] Optional, such as Figure 4 As shown, when the refrigeration unit is a refrigerator, light strips 201 can be installed on each shelf inside the refrigerator. When the electronic device detects that the user has put food into a shelf, the electronic device controls the light strip 201 of that shelf to light up to remind the user.

[0193] Figure 5 This is a schematic diagram of the shooting field of view of the first camera provided in an embodiment of this application, as shown below. Figure 5 As shown, taking a refrigerator as an example, each shelf in the refrigerator is not transparent. When the refrigerator door is opened, the camera can capture the field of vision of the shelf edge area.

[0194] In the above technical solution, by using a first camera installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf to capture the first image, the field of view of the refrigerator shelf area can be accurately obtained, avoiding missed detection.

[0195] In one embodiment of this application, the method further includes:

[0196] Control the feedback component corresponding to the target shelving area to output feedback information.

[0197] Optionally, the feedback component can be a light-emitting component installed in the refrigerator shelf area, including LED light strips, gas discharge tubes, quantum dots, phosphors, etc.; the feedback component can also be a voice component installed in the refrigerator shelf area, including voice assistants, smart speakers, etc.

[0198] Optionally, when the feedback component is a light-emitting component, such as an LED light strip, when the user places food into the shelf area, the electronic device can control the LED light strip corresponding to the destination shelf area to brighten, so as to remind the user that the food has been placed into the destination shelf area.

[0199] Optionally, when the feedback component is a voice component, such as a voice assistant, when the user places the food into the shelf area, the electronic device can control the voice assistant to output voice feedback information to inform the user that the food has been placed into the target shelf area.

[0200] In the above technical solution, the user experience can be improved and the practicality can be enhanced by controlling the feedback component corresponding to the target shelving area to output feedback information.

[0201] Figure 6 This is a schematic diagram of a food ingredient placement identification device provided in this application, such as... Figure 6 As shown, the food ingredient placement recognition device 60 includes an image acquisition unit 601, a detection and recognition unit 602, a first determination unit 603, and a second determination unit 604.

[0202] Image acquisition unit 601 is used to acquire multiple frames of first image;

[0203] The detection and recognition unit 602 is used to detect and recognize hands and food in the multiple frames of first images respectively, and to obtain the detection and recognition results of hands and food in each first image in the multiple frames of first images;

[0204] The first determining unit 603 is used to determine whether the user's action is to put food into the shelf based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first image. The position where the food is put into the shelf includes the edge of the shelf or a non-shelf edge.

[0205] The second determining unit 604 is used to determine the target shelf area where the food is placed, based on the target image of the bounding box of the hand holding the food in the multi-frame first image, when the user's action is determined to be putting food in.

[0206] Optionally, the hand and food detection and identification results include one of the following:

[0207] The bounding box of holding ingredients;

[0208] The bounding box of the hand;

[0209] The bounding box of the ingredients;

[0210] null.

[0211] Optionally, the first determining unit 603 is used to:

[0212] If the hand and food detection and recognition results corresponding to each first image in the multi-frame first image meet the first preset condition, then it is determined that the user's action is not putting in food.

[0213] The first preset conditions include:

[0214] In the hand and food detection and recognition results corresponding to each of the multiple first images, if there are N consecutive frames of hand and food detection and recognition results, the results will be one of the following:

[0215] Empty or food bounding box, hand bounding box, empty;

[0216] The bounding box of the empty space or food, the bounding box of the hand, the bounding box of the hand;

[0217] The bounding box of an empty or food item, the bounding box of the first hand holding the food item, the bounding box of the second hand holding the food item, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food item and the Y value of the bounding box of the second hand holding the food item is greater than or equal to a first value.

[0218] Optionally, the first determining unit 603 is used to:

[0219] If the hand and food detection and recognition results corresponding to each first image in the multi-frame first image meet the second preset condition, then it is determined that the user's action is to put in food.

[0220] The second preset condition includes:

[0221] Among the hand and food detection and recognition results corresponding to the multiple first images, the hand and food detection and recognition results of N consecutive images are one of the following:

[0222] Empty or bounding box of ingredients, bounding box of hand holding ingredients, empty;

[0223] The bounding box of an empty or food item, the bounding box of a hand holding a food item, and the bounding box of the food item.

[0224] The bounding box of an empty space or food ingredient, the bounding box of a hand holding food ingredient, and the bounding box of a hand.

[0225] The bounding box of an empty space or food item, the bounding box of a hand holding food item, the bounding box of a hand, and the bounding box of a food item.

[0226] The bounding box of an empty or food item, the bounding box of the first hand holding the food item, the bounding box of the second hand holding the food item, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food item and the Y value of the bounding box of the second hand holding the food item is less than or less than or equal to a first value.

[0227] Optionally, the second determining unit 604 is used for:

[0228] An image segmentation algorithm is used to segment the shelf area in the target image to obtain the shelf area segmentation result corresponding to the target image;

[0229] Based on the Y value of the bounding box of the hand holding the food in the target image and the segmentation result of the shelf area corresponding to the target image, the target shelf area where the food is placed is determined.

[0230] The target image is the last frame of the first image of the multi-frames before the ID of the same hand disappears, and the image of the bounding box of the hand holding the food is the result of the hand and food detection and recognition.

[0231] Optionally, the first image is an image captured by the first camera, or is obtained by processing the video captured by the first camera. The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf. The first camera is used to capture images of the shelf area after the door of the refrigeration unit is opened.

[0232] Optionally, the device further includes a reminder unit for:

[0233] Control the feedback component corresponding to the target shelving area to output feedback information.

[0234] In the above technical solution, the first image of the refrigeration device is acquired by the image acquisition unit, and the hand and food are detected and identified by the detection and recognition unit on multiple frames of the first image. The first determination unit determines whether the user's action is to put in food, and the second determination unit determines the shelf area where the food is to be put in. Based on the hand trajectory, it is first determined whether the hand action is to put in or take out food. If it is identified that food is to be put in, the shelf area where the user put in is further determined. This can improve the recognition accuracy, avoid false detection, and reduce the processing volume.

[0235] In this embodiment, the food placement recognition device 60 can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or any other device besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment does not specifically limit the specific type of device.

[0236] In this embodiment, the food placement recognition device 60 can be a device with an operating system. This operating system can be Microsoft Windows, Android, iOS, or other possible operating systems; this embodiment does not impose any specific limitations.

[0237] The food placement recognition device 60 provided in this embodiment can achieve… Figures 1 to 5 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0238] Figure 7 This is a schematic diagram of a food ingredient placement recognition system provided in an embodiment of this application, such as... Figure 7 As shown, the food placement recognition system 70 includes: a refrigeration unit 701, a first camera 702, and an electronic device 703. The first camera 702 is installed in the center of the top of the inner liner of the refrigeration unit 701 and is tilted towards the shelf. The first camera 702 is used to capture images of the shelf area after the door of the refrigeration unit 701 is opened.

[0239] The electronic device 703 includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the food placement recognition method as described in the foregoing method embodiments, which will not be repeated here.

[0240] It is worth noting that the electronic device 703 can be installed on the cooling device 701 or as a stand-alone device.

[0241] In the above technical solution, the first camera 702 captures the shelf area of ​​the refrigeration device 701 after the door is opened, and the electronic device 703 identifies the food placed into the refrigeration device 701 by processing the image captured by the first camera 702. The recognition accuracy can be improved by recognizing the hand and the food.

[0242] In some embodiments, such as Figure 8 As shown, the electronic device 703 in this embodiment includes a processor 801, a memory 802, and a computer program stored in the memory 802 and run on the processor 801. When the program is executed by the processor 801, it implements the various processes of the above-mentioned food placement identification method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0243] It should be noted that the electronic device in the embodiments of this application may be mobile or non-mobile.

[0244] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described food ingredient placement identification method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0245] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0246] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-mentioned food ingredient placement recognition method.

[0247] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-mentioned food placement recognition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0248] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0249] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0250] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0251] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0252] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0253] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for identifying food ingredients placed in a container, characterized in that, include: Acquire the first image of multiple frames; The hands and food ingredients are detected and identified in the multiple frames of the first image, and the detection and identification results of the hands and food ingredients in each first image are obtained. Based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first image, it is determined whether the user's action is to put food in, and the position where the food is put in includes the edge of the shelf or a non-shelf edge. If the user's action is determined to be placing food, the target shelf area for placing the food is determined based on the target image of the bounding box of the hand holding the food in the multi-frame first image, including the hand and food detection and recognition results.

2. The food ingredient placement identification method according to claim 1, characterized in that, The hand and food detection and identification results include one of the following: The bounding box of holding ingredients; The bounding box of the hand; The bounding box of the ingredients; null.

3. The food ingredient placement identification method according to claim 2, characterized in that, The step of determining whether the user's action is to put food in the food based on the hand and food detection and recognition results corresponding to each of the multiple first images includes: If the hand and food detection and recognition results corresponding to each first image in the multi-frame first image meet the first preset condition, then it is determined that the user's action is not putting in food. The first preset conditions include: In the hand and food detection and recognition results corresponding to each of the multiple first images, if there are N consecutive frames of hand and food detection and recognition results, the results will be one of the following: Empty or food bounding box, hand bounding box, empty; The bounding box of the empty space or food, the bounding box of the hand, the bounding box of the hand; The bounding box of an empty or food item, the bounding box of the first hand holding the food item, the bounding box of the second hand holding the food item, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food item and the Y value of the bounding box of the second hand holding the food item is greater than or equal to a first value.

4. The food placement recognition method according to claim 2, wherein determining whether the user's action is placing food based on the hand and food detection and recognition results corresponding to each of the multiple first images includes: If the hand and food detection and recognition results corresponding to each first image in the multi-frame first image meet the second preset condition, then the user's action is determined to be putting in food. The second preset condition includes: Among the hand and food detection and recognition results corresponding to the multiple first images, the hand and food detection and recognition results of N consecutive images are one of the following: Empty or bounding box of ingredients, bounding box of hand holding ingredients, empty; The bounding box of an empty or food item, the bounding box of a hand holding a food item, and the bounding box of the food item. The bounding box of an empty space or food ingredient, the bounding box of a hand holding food ingredient, and the bounding box of a hand. The bounding box of an empty space or food item, the bounding box of a hand holding food item, the bounding box of a hand, and the bounding box of a food item. The bounding box of an empty or food item, the bounding box of the first hand holding the food item, the bounding box of the second hand holding the food item, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food item and the Y value of the bounding box of the second hand holding the food item is less than or less than or equal to a first value.

5. The food ingredient placement identification method according to claim 1, characterized in that, The detection and recognition results of hands and food ingredients in the multi-frame first images include the target image of the bounding box of the hand holding the food ingredient, determining the target shelf area where the food ingredient is placed, including: An image segmentation algorithm is used to segment the shelf area in the target image to obtain the shelf area segmentation result corresponding to the target image; Based on the Y value of the bounding box of the hand holding food in the target image and the segmentation result of the shelf area corresponding to the target image, the target shelf area where the food is placed is determined. The target image is the last frame of the first image of the multi-frames before the ID of the same hand disappears, and the image of the bounding box of the hand holding the food is the result of the hand and food detection and recognition.

6. The food ingredient placement identification method according to claim 1, characterized in that, The first image is an image captured by the first camera, or obtained by processing the video captured by the first camera. The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf. The first camera is used to take pictures of the shelf area after the door of the refrigeration unit is opened.

7. The food ingredient placement identification method according to any one of claims 1-6, characterized in that, The method further includes: Control the feedback component corresponding to the target shelving area to output feedback information.

8. A food ingredient placed in a recognition device, characterized in that, include: An image acquisition unit is used to acquire multiple frames of the first image; The detection and recognition unit is used to detect and recognize hands and food in the multiple frames of first images respectively, and to obtain the detection and recognition results of hands and food in each first image in the multiple frames of first images; The first determining unit is used to determine whether the user's action is to put food into the shelf based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first image. The location where the food is put into the shelf includes the edge of the shelf or a location other than the edge of the shelf. The second determining unit is used to determine the target shelf area where the food is placed, based on the target image of the bounding box of the hand holding the food in the multi-frame first image, when the user's action is determined to be putting food in.

9. A food ingredient placement recognition system, characterized in that, include: The cooling unit, the first camera, and electronic equipment, among which, The first camera is installed at the top center of the inner liner of the refrigeration unit and tilted towards the shelf. The first camera is used to photograph the shelf area after the door of the refrigeration unit is opened. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the food placement recognition method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the food placement recognition method as described in any one of claims 1-7.

Images