Identification method, device and system for putting food materials in two hands and storage medium

Abstract

The invention discloses a recognition method, device and system for putting food materials in two hands and a storage medium, and belongs to the technical field of household appliances. The method comprises the following steps: acquiring multiple frames of first images; performing hand and food material detection and recognition on the multiple frames of first images to obtain hand and food material detection and recognition results corresponding to the first images in the multiple frames of first images; according to the hand and food material detection and recognition results corresponding to the first images in the multiple frames of first images, tracking the two-hand actions of the user, and determining whether the action of at least one hand in the two-hand actions of the user is food material putting or not; and under the condition that at least one of the actions of the two hands of the user is to put the food materials, determining a target shelf area in which the two hands put the food materials on the basis of a target image of a bounding box in which the hand and food material detection and recognition result in the multi-frame first image comprises the food material taking by the hand. By detecting the hands and the food materials and tracking the actions of the two hands of the user, accurate recognition of the food materials put into the two hands is achieved.

Description

Technical Field

[0001] This application belongs to the field of household appliance technology, and in particular relates to a method, device, system and storage medium for identifying food ingredients placed by both hands. Background Technology

[0002] In related technologies, refrigerator food retrieval and placement recognition generally uses image comparison technology for static recognition. Each shelf needs to be equipped with a camera. These food retrieval and placement recognition methods have problems such as high data collection costs, food that is close to the camera may be obstructed from the view and cannot be captured, and inaccurate recognition results, which bring many inconveniences to the practical application of refrigerator food recognition. Summary of the Invention

[0003] This application discloses a method, device, system, and storage medium for identifying food placed by both hands, in order to solve the problems of high acquisition cost in refrigerator food identification scenarios, inaccurate identification results due to potential obstruction of the field of view by food close to the camera, and inaccurate identification results.

[0004] Firstly, this application provides a method for recognizing food placed by both hands, the method comprising:

[0005] Acquire the first image of multiple frames;

[0006] The hands and food ingredients are detected and identified in the multiple frames of the first image, and the detection and identification results of the hands and food ingredients in each first image are obtained.

[0007] Based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first image, the user's hand movements are tracked to determine whether there is at least one hand movement for putting food in the food.

[0008] If at least one of the user's hand movements involves placing food, the target shelf area for placing food is determined based on the target image of the bounding box of the hand holding the food in the multi-frame first image, which includes the hand and food detection and recognition results.

[0009] In the above technical solution, by acquiring multiple frames of first images, detecting and recognizing hands and food in each frame, and tracking the user's hand movements based on the detection and recognition results of hands and food in each first image, it is determined whether at least one hand is engaged in placing food. If at least one hand is engaged in placing food, the target shelf area for placing food is determined based on the target image of the bounding box of the hand holding food in the detection and recognition results of hands and food in the multiple frames of first images. By tracking the user's hand movements, accurate recognition of placing food is achieved. Furthermore, determining the placement area only after confirming that at least one hand is engaged in placing food avoids unnecessary judgment processes and improves recognition accuracy.

[0010] According to the identification method for food ingredients placed by both hands in this application, the multiple first images are captured by a first camera, or obtained by processing the video captured by the first camera. The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf. The inner liner is provided with multiple non-transparent shelves.

[0011] In the above technical solution, the first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf, so that the first camera can accurately capture the shelf area after the door of the refrigeration unit is opened, avoiding missed detection.

[0012] According to the method for identifying food ingredients placed by both hands in this application, the detection and identification results of the hands and food ingredients include at least one of the following:

[0013] The bounding box of holding ingredients;

[0014] The bounding box of the hand;

[0015] null.

[0016] In the above technical solution, by classifying the recognition results of the first image into categories such as hand, hand holding food, and empty space, the recognition results can be reflected more clearly, which is convenient for subsequent recognition and judgment.

[0017] According to the method for recognizing food placed with both hands in this application, the step of tracking the user's hand movements based on the hand and food detection and recognition results corresponding to each of the multiple first images, and determining whether there is a hand movement indicating that at least one hand is placing food, includes:

[0018] If the hand and food detection and recognition results corresponding to the multi-frame first image meet the first preset condition, then it is determined that at least one of the user's hand movements is putting food into the container.

[0019] The first preset conditions include:

[0020] Among the hand and food detection and recognition results corresponding to the multi-frame first image, if there are three consecutive frames, the hand and food detection and recognition results are one of the following:

[0021] Empty; the bounding box of the hand holding the ingredients is empty.

[0022] Empty, the bounding box of the first hand holding the food, the bounding box of the second hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food and the Y value of the bounding box of the second hand holding the food is less than or less than or equal to the first value;

[0023] Empty, the bounding box of the hand holding the ingredients, the bounding box of the hand.

[0024] In the above technical solution, by setting a first preset condition, it is determined whether the user's hand movements include at least one hand movement of putting food into the container. When the hand and food detection and recognition results of each first image in multiple first images meet the first preset condition in N consecutive images, it is determined that the user's hand movements include at least one hand movement of putting food into the container, which can improve the recognition accuracy.

[0025] According to the food ingredient identification method of this application, the method further includes:

[0026] If the hand and food detection and recognition results corresponding to the first multi-frame first image do not meet the first preset condition, then it is determined that there is no action of at least one hand in the user's hand movements to put in food, and the process ends.

[0027] In the above technical solution, by setting a first preset condition to determine whether there is at least one hand in the user's hand movements that is putting in food, the process ends when there are N consecutive frames of hand and food detection and recognition results in the first images of multiple first images that do not meet the first preset condition, thereby improving recognition efficiency.

[0028] According to the method for recognizing food placed by both hands in this application, the step of determining the target shelf area for placing food by both hands based on the target image of the bounding box of the bounding box of the hands holding the food in the multi-frame first image includes:

[0029] Acquire the target image in the multi-frame first image, wherein the target image is the last frame of the hand and food detection and recognition results in the multi-frame first image before the ID of the same hand disappears, including the bounding box of the hand holding the food;

[0030] An image segmentation algorithm is used to segment the shelf area in the target image to obtain the shelf area segmentation result corresponding to the target image;

[0031] Based on the bounding box information of the hands holding food in the target image and the segmentation result of the shelf area corresponding to the target image, the target shelf area for placing the hands into the food is determined.

[0032] In the above technical solution, the shelf area in the target image is segmented by an image segmentation algorithm, which makes it easier to determine the specific location where the food is placed. By using the Y value of the bounding box of the hand holding the food in the target image and the segmentation result of the corresponding shelf area in the target image, the target shelf area where the hands put the food is determined, which reduces errors and improves recognition accuracy.

[0033] According to the method for recognizing food placed by both hands in this application, the step of determining the target shelf area for placing food by both hands based on the bounding box information of the hands holding food in the target image and the shelf area segmentation result corresponding to the target image includes:

[0034] If the hand and food detection and recognition results corresponding to the target image include two bounding boxes of hands holding food, the target shelf area for placing the food is determined based on the Y value of each bounding box of the hands holding food and the shelf area segmentation result corresponding to the target image.

[0035] If the detection and recognition results of the hand and food corresponding to the target image include a bounding box of a hand with a first ID and a bounding box of a hand holding food with a second ID, the target shelf area where the hand with the second ID puts the food is determined based on the Y value of the bounding box of the hand holding food with the second ID and the shelf area segmentation result corresponding to the target image.

[0036] In the above technical solution, based on the detection and recognition results of the hand and food corresponding to the target image and the segmentation results of the shelf area corresponding to the target image, the recognition of the user's hands putting food is realized with high recognition accuracy.

[0037] According to the food ingredient identification method of this application, the method further includes:

[0038] Control the reminder component corresponding to the target shelf area to output a prompt message.

[0039] In the above technical solution, controlling the output of prompt information by the reminder component corresponding to the target shelving area can improve the user experience and has better practicality.

[0040] Secondly, this application provides a recognition device for food ingredients placed by both hands, the device comprising:

[0041] An image acquisition unit is used to acquire multiple frames of the first image;

[0042] The detection and recognition unit is used to detect and recognize hands and food in the multiple frames of first images respectively, and to obtain the detection and recognition results of hands and food in each first image in the multiple frames of first images;

[0043] The first determining unit is used to track the user's hand movements based on the hand and food detection and recognition results corresponding to each first image in the multiple frames of first images, and to determine whether there is at least one hand movement of the user's hand movements that involves putting food in.

[0044] The second determining unit is used to determine the target shelf area for placing food by both hands, based on the target image of the bounding box of the hand holding the food in the multi-frame first image, when at least one of the user's hand movements is to put food in.

[0045] In the above technical solution, by acquiring multiple frames of first images, detecting and recognizing hands and food in each frame, and tracking the user's hand movements based on the detection and recognition results of hands and food in each first image, it is determined whether at least one hand is engaged in placing food. If at least one hand is engaged in placing food, the target shelf area for placing food is determined based on the target image of the bounding box of the hand holding food in the detection and recognition results of hands and food in the multiple frames of first images. By tracking the user's hand movements, accurate recognition of placing food is achieved. Furthermore, determining the placement area only after confirming that at least one hand is engaged in placing food avoids unnecessary judgment processes and improves recognition accuracy.

[0046] Thirdly, this application provides a recognition system for food ingredients placed by both hands, comprising: a refrigeration device, a first camera, and an electronic device, wherein...

[0047] The refrigeration unit has multiple non-transparent shelves inside its casing;

[0048] The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and is tilted towards the shelf.

[0049] The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for recognizing food placed by both hands as described in the first aspect.

[0050] In the above technical solution, by installing a first camera on the refrigeration device and setting multiple non-transparent shelves, the electronic device can track the user's hand movements, accurately identify the food being placed in the hands, and determine the placement area only after confirming that at least one of the user's hands is placing food, which can avoid unnecessary judgment processes and improve recognition accuracy.

[0051] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the identification method for food ingredients placed by both hands as described in the first aspect above.

[0052] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the identification method for food ingredients placed by both hands as described in the first aspect.

[0053] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the recognition method for food ingredients placed by both hands as described in the first aspect above.

[0054] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0055] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0056] Figure 1 This is a flowchart illustrating the method for recognizing food ingredients placed by both hands, as provided in an embodiment of this application.

[0057] Figure 2 This is a schematic diagram of the installation position of the first camera provided in an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of the shooting field of view of the first camera provided in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram illustrating the area of ​​the shelf where food items are placed, provided in an embodiment of this application.

[0060] Figure 5 This is a schematic diagram of the structure of the food identification device for placing food with both hands provided in the embodiments of this application;

[0061] Figure 6 This is a schematic diagram of the structure of the food identification system for placing ingredients with both hands, provided in an embodiment of this application;

[0062] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0063] Explanation of reference numerals in the attached figures:

[0064] 201: LED strip; 50: Recognition of food being placed in by both hands; 510: Image acquisition unit;

[0065] 520: Detection and recognition unit; 530: First determination unit; 540: Second determination unit;

[0066] 60: Food ingredient recognition system (recognition system for hands-on placement); 601: Refrigeration unit; 602: First camera;

[0067] 603: Electronic device; 701: Processor; 702: Memory. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0069] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0070] The following description, in conjunction with the accompanying drawings, details the identification method, apparatus, system, and storage medium for food ingredients placed by both hands provided in this application, through specific embodiments and application scenarios.

[0071] The food placement recognition method provided in this application embodiment can be implemented by an electronic device or a functional module or entity within an electronic device that can realize the food placement recognition method. The electronic devices mentioned in this application embodiment include, but are not limited to, controllers / processors in refrigeration devices, edge servers, backend servers, cloud servers, mobile phones, tablets, computers, cameras, and wearable devices. The food placement recognition method provided in this application embodiment will be described below using an electronic device as the implementation subject.

[0072] Figure 1 This is a flowchart illustrating the method for recognizing food ingredients placed by both hands, as provided in an embodiment of this application. Figure 1 As shown, the identification method for placing the food with both hands includes steps 110, 120, 130 and 140.

[0073] Step 110: Obtain the first image of multiple frames.

[0074] In this embodiment of the application, the electronic device acquires multiple frames of first images of the cooling device. The first images may be images acquired by a acquisition device installed on the cooling device, or images obtained by processing the first video acquired by the acquisition device through the electronic device.

[0075] The data acquisition device can be a camera or a sensor capable of generating images or videos. It should be understood that, in order to identify food items placed in the refrigeration unit, the field of view of the acquisition device includes the shelf area after the refrigeration unit door is opened. That is, the multiple first images are obtained by capturing images of the shelf area after the refrigeration unit door is opened.

[0076] It should be noted that the refrigeration device in the embodiments of this application includes refrigerators, freezers, and other devices that can refrigerate and have a cabinet and door. The refrigeration device in the embodiments of this application can also be replaced by a cargo container, all of which are within the protection scope of this application.

[0077] The following text uses a refrigerator as an example of a refrigeration device.

[0078] In some embodiments, the multiple first images are captured by a first camera, or obtained by processing the video captured by the first camera. The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf. The inner liner is provided with multiple non-transparent shelves.

[0079] The data acquisition device is the first camera. The field of view of the first camera is the first field of view, which can be the view after the refrigerator door is opened, including the shelf area after the refrigeration unit door is opened.

[0080] The video captured by the first camera is called the first video. The first video can be a video of the shelf area after the door of the refrigeration unit is opened, captured by the first camera during the process of putting or taking food into or out of the refrigeration unit.

[0081] It should be noted that after the electronic device acquires the first video, it performs frame-by-frame processing on the first video to obtain multiple frames of the first field of view image. The first image is one frame of the multiple frames of the first field of view image.

[0082] Optionally, the first camera can be a wide-angle camera, infrared camera, high-definition camera, panoramic camera, wireless camera, night vision camera, smart camera, etc. This application embodiment does not limit the type of the first camera, and the type of camera can be selected by comprehensively considering cost and performance.

[0083] Optionally, the first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf, so that the first camera can accurately capture the shelf area after the door of the refrigeration unit is opened, avoiding missed detection.

[0084] In this embodiment of the application, the refrigeration device has multiple non-transparent shelves inside its casing.

[0085] If the shelves inside the refrigeration unit are transparent, it may cause reflection or refraction, affecting the shooting effect of the first camera. In order to prevent interference from target detection and recognition in the field of view, the shelves of the refrigerator in this embodiment are set to be non-transparent. Non-transparent shelves can provide better contrast, reduce reflection and interference, improve the clarity and quality of the captured image, and make it easier for the first camera to distinguish between hands, food and background, thereby improving the accuracy and efficiency of recognition.

[0086] Figure 2 This is a schematic diagram of the installation position of the first camera provided in an embodiment of this application, as shown below. Figure 2 As shown, the refrigeration device can be a refrigerator, and the first camera 602 can be installed in the middle of the top of the inner liner of the refrigerator and tilted towards the shelf. The first camera acquires images or videos within the field of view of the shelf area after the refrigerator door is opened.

[0087] Optional, such as Figure 2 As shown, when the refrigeration unit is a refrigerator, light strips 201 can be installed on each shelf inside the refrigerator. When the electronic device detects that the user has put food into a shelf, the electronic device controls the light strip 201 of that shelf to light up to remind the user.

[0088] Figure 3 This is a schematic diagram of the shooting field of view of the first camera provided in an embodiment of this application, as shown below. Figure 3As shown, taking a refrigerator as an example, each shelf in the refrigerator is not transparent. When the refrigerator door is opened, the first camera can capture the field of view of the shelf edge area.

[0089] It is worth noting that the electronic device in the embodiments of this application can be any device that can meet the needs of data transmission and data processing (such as data calculation, deletion or storage, food identification and positioning). For example, it can be the controller / processor of the refrigeration device, or a functional module or entity of the controller / processor of the refrigeration device, or a terminal device, edge server, back-end server, cloud server or other hardware device that can communicate with the controller / processor of the refrigeration device, the acquisition device (first camera).

[0090] In one embodiment, the first camera sends the captured first video to the refrigerator motherboard (which has a controller / processor). The refrigerator motherboard uploads the first video to the cloud server in real time. The cloud server performs frame-by-frame processing on the first video to obtain multiple first images. Subsequent steps are then executed by the cloud server.

[0091] In one embodiment, the first camera sends the captured first video to the refrigerator motherboard, which then performs frame-by-frame processing on the first video to obtain multiple first images. Subsequent steps are executed by the controller / processor on the refrigerator motherboard.

[0092] In one embodiment, the first camera uploads the captured first video to a cloud server, which then performs frame-by-frame processing on the first video to obtain multiple first images. Subsequent steps are then executed by the cloud server.

[0093] In one embodiment, the first camera uploads the captured first video to an edge server. The edge server performs frame-by-frame processing on the first video to obtain multiple first images, and then uploads the multiple first images to a cloud server. Subsequent steps are then performed by the cloud server.

[0094] Step 120: Detect and identify hands and food ingredients in the multiple frames of the first image respectively, and obtain the detection and identification results of hands and food ingredients in each first image in the multiple frames of the first image;

[0095] It should be noted that when the electronic device detects and recognizes multiple frames of the first image, it will detect and recognize the hand and food in the multiple frames of the first image separately. The recognition result of the electronic device includes the hand holding the food, the hand, or one of them in the air.

[0096] Optionally, when a hand holding food appears in the first image, the electronic device will mark the hand holding food with a rectangle; when a hand appears in the first image, the electronic device will mark the hand with a rectangle; when there is neither a hand nor food in the first image, the electronic device will not make a mark, and the recognition result will be empty.

[0097] In this step, taking a refrigerator as an example, the refrigerator shelf area can be divided. The electronic device can comprehensively determine which shelf the user has placed the food on based on the detection and recognition results of the hand and food in each of the first images in multiple frames of the first image.

[0098] Step 130: Based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first image, track the user's hand movements to determine whether there is at least one hand movement for putting food into the food.

[0099] It is worth noting that when a hand or a hand plus food is detected in the first image, it indicates that the user may have put food in the food.

[0100] In order to recognize the action of putting food into the container with both hands, this embodiment of the application tracks the user's hand movements based on the hand and food detection and recognition results corresponding to each first image in multiple first images, thereby determining whether there is at least one hand action of putting food into the container.

[0101] When tracking a user's hands, the system comprehensively analyzes the first images from multiple preceding and following frames. When a hand appears in the current frame or is holding food, the earliest appearing hand is assigned an ID of 1, and the subsequent hand is assigned an ID of 2. Different hands are distinguished by their IDs. Tracking the two hands does not interfere with each other.

[0102] If it is determined that at least one of the user's hand gestures involves placing food, the system will continue to determine which shelf of the refrigeration unit the user placed the food on.

[0103] The electronic device will determine whether the detection and recognition results of the hands and food in each of the first images in the multi-frame first images meet the preset condition that the action of at least one hand is to put the food in.

[0104] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple first images, there are N consecutive images where the hand and food detection and recognition results are empty, the bounding box of the hand holding the food, and empty in sequence. This indicates that the user has put the food in the shelf with at least one hand.

[0105] Optionally, the bounding boxes of hands holding food detected in a first image include: bounding boxes of hands holding food with different IDs. If the hand and food detection and recognition results of multiple frames of the first image contain N consecutive frames of images with the hand and food detection and recognition results being empty, two bounding boxes of hands holding food with different IDs, and empty, it indicates that the user has put food into the food with both hands.

[0106] Optionally, the bounding box of a hand holding food detected in a first image includes: a bounding box of a hand and a bounding box of a hand holding food. If the hand and food detection and recognition results of multiple frames of the first image show N consecutive frames with the following results: empty, a bounding box of a hand and a bounding box of a hand holding food, and empty, it indicates that one of the user's hands has performed the action of putting food into the container.

[0107] Optionally, the bounding box detected in a first image for a hand holding food includes: a bounding box for a hand holding food. If, in the hand and food detection and recognition results of multiple frames of the first image, there are N consecutive frames where the hand and food detection and recognition results are empty, a bounding box for a hand holding food, and empty, it indicates that one of the user's hands has performed the action of putting food into the container.

[0108] In this embodiment of the application, the value of N is greater than or equal to 3.

[0109] Optionally, the preset condition can be that in the hand and food detection and recognition results corresponding to multiple frames of the first image, there are N consecutive frames where the hand and food detection and recognition results are empty, the bounding box of the first hand holding food, and the bounding box of the second hand holding food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding food and the Y value of the bounding box of the second hand holding food is less than or equal to a first value. This indicates that at least one of the user's hands has performed the action of putting food in, but not completely putting the food in. If the bounding box of the same hand holding food appears in two consecutive frames of the first image, it indicates that the hand action is to pick up food. Whether it is putting in or taking out needs to be determined based on the change in the Y value between the bounding boxes. In this embodiment, if the absolute value of the difference in Y values ​​is less than or equal to the first value, that is, the change in Y values ​​is small, it is determined that the user's action is to put food in by bringing the food closer to the shelf area. If the absolute value of the difference in Y values ​​is greater than or equal to the first value, that is, the change in Y values ​​is large, it is determined that the user's action is to take food away from the shelf area.

[0110] Optionally, the bounding boxes of the first / second hand holding the food include: two bounding boxes of hands holding food with different IDs, or a bounding box of one hand and a bounding box of one hand holding food. It should be noted that when comparing the bounding boxes of the first and second hands holding food, the comparison should be made for the same hand in both frames of the first image.

[0111] Understandably, if for the same hand, the detection and recognition results of the hand and food in three consecutive frames of the first image are empty, the bounding box of the hand holding the food, and the bounding box of the hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the hand holding the food in the third frame and the Y value of the bounding box of the hand holding the food in the second frame is less than or equal to the first value, then it means that the action of the hand is to put the food in.

[0112] It should be noted that the Y value of the bounding box can be a coordinate value of the bounding box. If the bounding box is given in the form of center point coordinates (x_center, y_center) and width and height, then the Y value of the bounding box represents the vertical coordinate of the center point of the bounding box in the image.

[0113] Optionally, if the bounding box is given in the form of top-left corner coordinates (x1, y1) and bottom-right corner coordinates (x2, y2), then the Y value of the bounding box refers to the vertical coordinate of the top-left corner of the bounding box.

[0114] It is worth noting that the first value can be a number set according to the actual situation.

[0115] Optionally, the preset condition can be that the hand and food detection and recognition results of N consecutive frames of the first image are empty, the bounding box of the hand holding the food, and the bounding box of the hand in sequence. At this time, it can be determined that at least one of the user's hands has performed the action of putting the food into the shelf. After the food is put in, the hand has not completely left the collection field of view of the acquisition device. Therefore, the bounding box of the hand exists in the first image.

[0116] It should be noted that the bounding box of the hand holding the food can be two bounding boxes of hands holding food with different IDs, or it can be a bounding box of one hand and a bounding box of one hand holding food. The bounding box of the hand can be two bounding boxes of hands with different IDs, or it can be a bounding box of one hand.

[0117] It should be noted that the embodiments of this application only identify when the user puts in food. When the user's hand movements do not include at least one hand movement of putting in food, that is, when neither hand movement is putting in food, the process ends.

[0118] Optionally, if the hand and food detection and recognition results of the first images in multiple frames contain consecutive N frames where the hand and food detection and recognition results are empty, a bounding box of the hand, and empty, then the user's action may be taking their hand out of the refrigerator or putting their hand into the refrigerator to retrieve food, then the process ends.

[0119] Here, the bounding box of a hand includes the bounding box of one hand, or the bounding box of both hands.

[0120] Optionally, if the hand and food detection and recognition results of the first images in multiple frames contain N consecutive frames where the hand and food detection and recognition results are empty, a bounding box of the hand, and a bounding box of the hand, then the user's action may be taking their hand out of the refrigerator or putting their hand into the refrigerator to retrieve food, then the process ends.

[0121] Optionally, if the hand and food detection and recognition results of the first images in multiple frames contain N consecutive frames where the hand and food detection and recognition results are empty, the bounding box of the first hand holding food, and the bounding box of the second hand holding food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding food and the Y value of the bounding box of the second hand holding food is greater than or equal to the first value, then the user's action may be taking food out of the refrigerator, and the process ends.

[0122] Step 140: If at least one of the user's hand movements is to put food in, the target shelf area for putting food in is determined based on the target image of the bounding box of the hand holding the food in the multi-frame first image, which includes the hand and food detection and recognition results.

[0123] In this step, if at least one of the user's hand gestures involves placing food, the electronic device will further determine the specific location where the food is placed.

[0124] Optionally, the electronic device uses an image segmentation algorithm to divide the shelf area and adds an index.

[0125] In this embodiment, the target image is the bounding box of the hand holding food in the hand and food detection and recognition results of the multi-frame first image. Further, the target image is the last frame of the multi-frame first image before the same hand disappears, where the hand and food detection and recognition results show the bounding box of the hand holding food.

[0126] As the user places the food, the electronic device detects multiple frames of the bounding box of the hand holding the food. The position of the food placed in the last frame before the same hand disappears, which includes the bounding box of the hand holding the food, is the final position of the food.

[0127] In this embodiment of the application, the target shelf area where the food is finally placed can be determined by judging the correspondence between the Y value of the bounding box of the hand holding the food in the target image and the shelf area.

[0128] Optionally, light strips can be installed along the edge of each shelf. When the electronic device detects that food has been placed on a shelf, it controls the light strip on that shelf to light up to remind the user.

[0129] The food placement recognition method provided in this application acquires multiple frames of first images, detects and recognizes hands and food in each frame, and tracks the user's hand movements based on the detection and recognition results of hands and food in each first image. It determines whether at least one hand is involved in placing food. If at least one hand is involved, the method determines the target shelf area for placing food by hand, based on the target image including the bounding box of the hand holding food in the multi-frame first images. By tracking the user's hand movements, accurate recognition of food placement by hand is achieved. Furthermore, determining the placement area only after confirming that at least one hand is involved avoids unnecessary judgment processes and improves recognition accuracy.

[0130] In some embodiments, the hand and food detection and identification results include at least one of the following:

[0131] The bounding box of holding ingredients;

[0132] The bounding box of the hand;

[0133] null.

[0134] In this embodiment of the application, when the electronic device detects multiple frames of the first image, it will classify and identify them according to whether the hand is holding food, the hand, or the empty space.

[0135] Optionally, when a hand appears in the first image, the electronic device marks the hand with a rectangle and assigns an ID to the hand.

[0136] Optionally, when a hand holding food appears in the first image, the electronic device marks the hand holding food with a rectangle.

[0137] It's important to note that electronic devices can determine whether food is being handled by hand based on the degree of overlap between the hand and the food. For example, they can calculate the intersection-union ratio (IUR) between the rectangles representing the hand and the food. The IUR is calculated by dividing the area of ​​their intersection by the area of ​​their union; the value ranges from 0 to 1, with a higher value indicating greater overlap. The electronic device determines whether food is being handled by hand by comparing the IUR to a preset threshold.

[0138] Optionally, the preset threshold can be set between 0 and 1 according to the actual situation. For example, if the preset threshold is set to 0.5, the intersection-union ratio between the rectangle of the hand and the rectangle of the food in the first image is 0.6, which is greater than the preset threshold. At this time, it is determined that the hand is holding the food, and the rectangle is used to mark the hand holding the food.

[0139] In one embodiment of this application, the rectangle of the hand used to calculate whether it is holding food can also be a triangle. When the electronic device detects that the hand and food overlap in the first image, it can mark the food with a rectangle. The device determines whether the food is being held by hand by comparing the intersection-union ratio (IU) between the rectangle of the food and the triangle of the hand with a preset threshold. For example, if the IU is 0.75, which is greater than the preset threshold of 0.5, the electronic device determines that the food is being held by hand and marks the food with a rectangle.

[0140] Optionally, when there is neither a hand nor food in the first image, the electronic device will recognize the image as empty and will not label the first image.

[0141] Optionally, the bounding box for holding the food includes one of the following:

[0142] A bounding box containing a hand holding ingredients;

[0143] A bounding box for a hand and a bounding box for a hand holding food;

[0144] The bounding boxes of two different IDs holding ingredients.

[0145] Optionally, the bounding box of the hand includes one of the following:

[0146] The bounding box of a hand;

[0147] Bounding boxes for two hands with different IDs.

[0148] The food ingredient identification method provided in this application embodiment allows the electronic device to classify the identification results of the first image into categories such as hands, hands holding food, and empty spaces, which can more clearly reflect the identification results and facilitate subsequent identification and judgment.

[0149] In some embodiments, the step of tracking the user's hand movements based on the hand and food detection and recognition results corresponding to each of the first images in the multiple first images, and determining whether there is a hand movement of at least one hand that involves placing food, includes:

[0150] If the hand and food detection and recognition results corresponding to the multi-frame first image meet the first preset condition, then it is determined that at least one of the user's hand movements is putting food into the container.

[0151] The first preset conditions include:

[0152] Among the hand and food detection and recognition results corresponding to the multi-frame first image, if there are three consecutive frames, the hand and food detection and recognition results are one of the following:

[0153] Empty; the bounding box of the hand holding the ingredients is empty.

[0154] Empty, the bounding box of the first hand holding the food, the bounding box of the second hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food and the Y value of the bounding box of the second hand holding the food is less than or less than or equal to the first value;

[0155] Empty, the bounding box of the hand holding the ingredients, the bounding box of the hand.

[0156] It is worth noting that when the electronic device determines whether the user's hand movements include at least one hand movement for placing food, it needs to make a judgment through multiple consecutive first images. If the hand and food detection and recognition results corresponding to each first image in the multiple first images meet the first preset condition, then it is determined that the user's hand movements include at least one hand movement for placing food.

[0157] Optionally, the first preset condition is that among the user's hand movements, there is a set of various possibilities where at least one hand is putting in food. The first preset condition includes that among the hand and food detection and recognition results corresponding to each of the first images in multiple first images, there are N consecutive images where the hand and food detection and recognition results are specific results in sequence.

[0158] Where N is a positive integer greater than 1.

[0159] The user's hand gestures include at least one hand action of placing food, including at least one of the following:

[0160] Both hands were used to place ingredients;

[0161] One hand is used to add ingredients, while the other hand is used to add ingredients.

[0162] Among them, "not placing food" includes: placing food and leaving the shelf area empty-handed; entering the shelf area empty-handed, preparing to take food, and leaving the shelf area with food in hand.

[0163] Because there are many possibilities that the hand might not contain food, the electronic device tracks and judges each hand separately when tracking the user's hands. If a hand disappears, its ID will be reassigned when the user's hand reappears in the first image.

[0164] Optionally, if the hand and food detection and recognition results of the first images in multiple frames contain N consecutive frames where the hand and food detection and recognition results are empty, the bounding box of the hand holding the food, and empty, it indicates that the user has put the food into the shelf with at least one hand.

[0165] Optionally, if the hand and food detection and recognition results of the first images in multiple frames contain consecutive N frames where the hand and food detection and recognition results are empty, bounding boxes of two hands with different IDs holding food, and empty, it indicates that the user has put food into the food with both hands.

[0166] Optionally, if the hand and food detection and recognition results of the first images in multiple frames contain N consecutive frames where the hand and food detection and recognition results are empty, a bounding box of a hand and a bounding box of a hand holding food, and empty, it indicates that the user's hand has performed the action of putting food into the container.

[0167] Optionally, if the hand and food detection and recognition results of the first images in multiple frames contain N consecutive frames where the hand and food detection and recognition results are empty, a bounding box of a hand holding food, and empty, it indicates that the user's hand has performed the action of putting food into the container.

[0168] Optionally, if the hand and food detection and recognition results of N consecutive frames of the first image contain empty frames, bounding boxes of hands holding food, and bounding boxes of hands, it indicates that the user has placed food in the food with at least one hand, and the hand has not completely left the field of view of the first camera after the food is placed in.

[0169] Optionally, if in the hand and food detection and recognition results corresponding to multiple first images there are N consecutive images where the hand and food detection and recognition results are successively empty, the bounding box of the first hand holding the food, the bounding box of the second hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food and the bounding box of the second hand holding the food is less than or equal to a first value, then it indicates that at least one of the user's hands performed the action of placing the food, but the food was not completely placed. In the next image after the food is placed, the bounding box of the hand holding the food will disappear, that is, the hand and food detection and recognition result in the next image after the food is placed will be empty.

[0170] Optionally, the bounding boxes of the first / second hand holding the food include: two bounding boxes of hands holding food with different IDs, or a bounding box of one hand and a bounding box of one hand holding food. It should be noted that when comparing the bounding boxes of the first and second hands holding food, the comparison should be made for the same hand in both frames of the first image.

[0171] Understandably, if for the same hand, the detection and recognition results of the hand and food in three consecutive frames of the first image are empty, the bounding box of the hand holding the food, and the bounding box of the hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the hand holding the food in the third frame and the Y value of the bounding box of the hand holding the food in the second frame is less than or equal to the first value, then it means that the action of the hand is to put the food in.

[0172] For an understanding of the Y-value of the bounding box, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0173] The food ingredient placement recognition method provided in this application embodiment determines whether the user's hand movements include at least one hand movement as food placement by setting a first preset condition. When the hand and food detection and recognition results of each first image in multiple first images meet the first preset condition in N consecutive images, it is determined that the user's hand movements include at least one hand movement as food placement, which can improve the recognition accuracy.

[0174] In some embodiments, the method further includes:

[0175] If the hand and food detection and recognition results corresponding to the first multi-frame first image do not meet the first preset condition, then it is determined that there is no action of at least one hand in the user's hand movements to put in food, and the process ends.

[0176] It is understood that if the hand and food detection and recognition results corresponding to the first multi-frame first image do not meet the first preset condition, then it is determined that there is no action of at least one hand in the user's hand movements to put in food, and the process ends.

[0177] Optionally, if the hand and food detection and recognition results of the first images in multiple frames contain N consecutive frames where the hand and food detection and recognition results are empty, a bounding box of the hand, and empty, then the user's action may be taking the hand out of the refrigerator or putting the hand into the refrigerator to take food. This does not meet the first preset condition, and the process ends.

[0178] Here, the bounding box of a hand includes the bounding box of one hand, or the bounding boxes of two hands with different IDs.

[0179] Optionally, if the hand and food detection and recognition results of the first images in multiple frames contain N consecutive frames where the hand and food detection and recognition results are empty, a bounding box of a hand, and a bounding box of a hand, then the user's action may be taking their hand out of the refrigerator or putting their hand into the refrigerator to retrieve food. If the first preset condition is not met, then the process ends.

[0180] Optionally, if among the hand and food detection and recognition results corresponding to multiple first images, there are N consecutive images where the hand and food detection and recognition results are empty, the bounding box of the first hand holding food, and the bounding box of the second hand holding food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding food and the Y value of the bounding box of the second hand holding food is greater than or equal to a first value, then the user's action may be taking food out of the refrigerator, which does not meet the first preset condition, and the process ends.

[0181] The food ingredient placement recognition method provided in this application embodiment determines whether at least one hand in the user's hand movements constitutes food placement by setting a first preset condition. When the hand and food ingredient detection and recognition results of each first image in multiple first images do not meet the first preset condition for N consecutive images, the process ends, which can improve recognition efficiency.

[0182] In some embodiments, the step of determining the target shelf area where the hands are placed into the food based on the detection and recognition results of the hands and food in the multi-frame first images includes the target image of the bounding box of the hands holding the food, and includes:

[0183] Acquire the target image in the multi-frame first image, wherein the target image is the last frame of the hand and food detection and recognition results in the multi-frame first image before the ID of the same hand disappears, including the bounding box of the hand holding the food;

[0184] An image segmentation algorithm is used to segment the shelf area in the target image to obtain the shelf area segmentation result corresponding to the target image;

[0185] Based on the bounding box information of the hands holding food in the target image and the segmentation result of the shelf area corresponding to the target image, the target shelf area for placing the hands into the food is determined.

[0186] Understandably, by using the image of the last frame of the first image before the ID of the same hand disappears, which is the bounding box of the hand holding the food, as the target image, the shelf area where the food is placed can be accurately determined.

[0187] After determining the target image, the electronic device can use an image segmentation algorithm to segment the shelf area in the target image, obtain the shelf area segmentation result corresponding to the target image, and add an index to each segmented shelf area.

[0188] Figure 4 This is a schematic diagram illustrating the purpose of placing food items into the shelf area provided in this application embodiment. Figure 4 As shown, taking a refrigerator as an example, Figure 4 It includes three fitted dividing lines for the refrigerator shelves, including red, green, and yellow dividing lines, which can be used to set the upper shelf area of ​​the refrigerator. Figure 4 The area between the upper and middle shelves of the refrigerator (index number 1, above the red dividing line) is the section with index number 1. Figure 4 The area between the red and green dividing lines (index number 2) is the area between the middle and lower shelves of the refrigerator. Figure 4 The index number between the green and yellow dividing lines is 3.

[0189] Optionally, the electronic device marks the hands holding food in the target image with rectangular boxes, such as... Figure 4 The red rectangle shown is the bounding box of the hand holding the food. Based on the Y-value of the bounding box of the hand holding the food in the target image (e.g., ...), ... Figure 4 The relationship between the ordinate of the top left corner of the green rectangle and the fitting dividing lines of the three refrigerator shelves is used to determine the shelf area where the food is placed, i.e., the shelf area where the food is placed.

[0190] For example, if the Y value is above the red dividing line, it is considered to be placed on the upper shelf; if the Y value is between the red and green dividing lines, it is considered to be placed on the middle shelf; if the Y value is between the green and yellow dividing lines, it is considered to be placed on the lower shelf.

[0191] The food placement recognition method provided in this application uses an image segmentation algorithm to segment the shelf area in the target image, which makes it easier to determine the specific location where the food is placed. By using the Y value of the bounding box of the hand holding the food in the target image and the segmentation result of the corresponding shelf area in the target image, the target shelf area for placing the food with both hands is determined, reducing errors and improving recognition accuracy.

[0192] In some embodiments, determining the target shelf area where the hands place the food based on the bounding box information of the hands holding the food in the target image and the shelf area segmentation result corresponding to the target image includes:

[0193] If the hand and food detection and recognition results corresponding to the target image include two bounding boxes of hands holding food, the target shelf area for placing the food is determined based on the Y value of each bounding box of the hands holding food and the shelf area segmentation result corresponding to the target image.

[0194] If the detection and recognition results of the hand and food corresponding to the target image include a bounding box of a hand with a first ID and a bounding box of a hand holding food with a second ID, the target shelf area where the hand with the second ID puts the food is determined based on the Y value of the bounding box of the hand holding food with the second ID and the shelf area segmentation result corresponding to the target image.

[0195] It is understood that if the hand and food detection and recognition results corresponding to the target image include two bounding boxes of hands holding food, then the target shelf area where the user's hands put the food is determined based on the positional relationship between the Y value of each bounding box of hands holding food and the fitted dividing line of the refrigerator shelf in the target image.

[0196] If the hand and food detection and recognition results corresponding to the target image include a bounding box of a hand with a first ID and a bounding box of a hand with a second ID holding food, that is, the user's action of one hand is to put food in, and the action of the other hand is not to put food in, then based on the Y value of the bounding box of the hand with the second ID holding food and the shelf area segmentation result corresponding to the target image, the target shelf area for the hand with the second ID to put food in is determined.

[0197] The food ingredient recognition method provided in this application realizes the recognition of users placing food ingredients with both hands based on the detection and recognition results of hands and food ingredients corresponding to the target image and the segmentation results of the shelf area corresponding to the target image, with high recognition accuracy.

[0198] In some embodiments, the method further includes:

[0199] Control the reminder component corresponding to the target shelf area to output a prompt message.

[0200] Optionally, the reminder component can be a light-emitting component installed in the refrigerator shelf area, including LED light strips, gas discharge tubes, quantum dots, phosphors, etc.; the reminder component can also be a voice component installed in the refrigerator shelf area, including voice assistants, smart speakers, etc.

[0201] Optionally, when the reminder component is a light-emitting component, such as an LED light strip, when the user places food into the shelf area, the electronic device can control the LED light strip corresponding to the destination shelf area to light up, so as to remind the user that the food has been placed into the destination shelf area.

[0202] Optionally, when the reminder component is a voice component, such as a voice assistant, when the user places the food into the shelf area, the electronic device can control the voice assistant to output voice prompts to inform the user that the food has been placed into the target shelf area.

[0203] The food ingredient placement recognition method provided in this application can improve the user experience and has better practicality by controlling the output of prompt information by the reminder component corresponding to the target shelf area.

[0204] The food ingredient recognition method provided in this application can be implemented by a food ingredient recognition device that recognizes food ingredients placed with both hands. This application uses the food ingredient recognition device that recognizes food ingredients placed with both hands as an example to illustrate the food ingredient recognition device provided in this application.

[0205] Figure 5 This is a schematic diagram of the structure of the food ingredient recognition device provided in this application embodiment, which identifies ingredients placed by both hands. Figure 5As shown, the identification device 50 for food ingredients placed by both hands includes: an image acquisition unit 510, a detection and identification unit 520, a first determination unit 530, and a second determination unit 540. Among them,

[0206] Image acquisition unit 510 is used to acquire multiple frames of first image;

[0207] The detection and recognition unit 520 is used to detect and recognize hands and food in the multiple frames of first images respectively, and obtain the detection and recognition results of hands and food in each first image in the multiple frames of first images;

[0208] The first determining unit 530 is used to track the user's hand movements based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first images, and to determine whether there is at least one hand movement of the user's hand movements that involves putting food in.

[0209] The second determining unit 540 is used to determine the target shelf area for placing food with both hands, based on the target image of the bounding box of the hand holding the food in the multi-frame first image, when at least one of the user's hand movements is to put food in.

[0210] The food placement recognition method provided in this application acquires multiple frames of first images, detects and recognizes hands and food in each frame, and tracks the user's hand movements based on the detection and recognition results of hands and food in each first image. It determines whether at least one hand is involved in placing food. If at least one hand is involved, the method determines the target shelf area for placing food by hand, based on the target image including the bounding box of the hand holding food in the multi-frame first images. By tracking the user's hand movements, accurate recognition of food placement by hand is achieved. Furthermore, determining the placement area only after confirming that at least one hand is involved avoids unnecessary judgment processes and improves recognition accuracy.

[0211] In some embodiments, the multiple first images are captured by a first camera, or obtained by processing the video captured by the first camera. The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf. The inner liner is provided with multiple non-transparent shelves.

[0212] In some embodiments, the hand and food detection and identification results include at least one of the following:

[0213] The bounding box of holding ingredients;

[0214] The bounding box of the hand;

[0215] null.

[0216] In some embodiments, the first determining unit 530 is configured to:

[0217] If the hand and food detection and recognition results corresponding to the multi-frame first image meet the first preset condition, then it is determined that at least one of the user's hand movements is putting food into the container.

[0218] The first preset conditions include:

[0219] Among the hand and food detection and recognition results corresponding to the multi-frame first image, if there are three consecutive frames, the hand and food detection and recognition results are one of the following:

[0220] Empty; the bounding box of the hand holding the ingredients is empty.

[0221] Empty, the bounding box of the first hand holding the food, the bounding box of the second hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food and the Y value of the bounding box of the second hand holding the food is less than or less than or equal to the first value;

[0222] Empty, the bounding box of the hand holding the ingredients, the bounding box of the hand.

[0223] In some embodiments, the first determining unit 530 is further configured to:

[0224] If the hand and food detection and recognition results corresponding to the first multi-frame first image do not meet the first preset condition, then it is determined that there is no action of at least one hand in the user's hand movements to put in food, and the process ends.

[0225] In some embodiments, the second determining unit 540 is configured to:

[0226] Acquire the target image in the multi-frame first image, wherein the target image is the last frame of the hand and food detection and recognition results in the multi-frame first image before the ID of the same hand disappears, including the bounding box of the hand holding the food;

[0227] An image segmentation algorithm is used to segment the shelf area in the target image to obtain the shelf area segmentation result corresponding to the target image;

[0228] Based on the bounding box information of the hands holding food in the target image and the segmentation result of the shelf area corresponding to the target image, the target shelf area for placing the hands into the food is determined.

[0229] In some embodiments, determining the target shelf area where the hands place the food based on the bounding box information of the hands holding the food in the target image and the shelf area segmentation result corresponding to the target image includes:

[0230] If the hand and food detection and recognition results corresponding to the target image include two bounding boxes of hands holding food, the target shelf area for placing the food is determined based on the Y value of each bounding box of the hands holding food and the shelf area segmentation result corresponding to the target image.

[0231] If the detection and recognition results of the hand and food corresponding to the target image include a bounding box of a hand with a first ID and a bounding box of a hand holding food with a second ID, the target shelf area where the hand with the second ID puts the food is determined based on the Y value of the bounding box of the hand holding food with the second ID and the shelf area segmentation result corresponding to the target image.

[0232] In some embodiments, the device further includes a reminder unit for controlling a reminder component corresponding to the target shelving area to output a reminder message.

[0233] The food identification device for hands placing ingredients in this embodiment can be an electronic device or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment does not specifically limit the device.

[0234] The food identification device for hands placing ingredients in this embodiment can be a device with an operating system. This operating system can be Microsoft Windows, Android, iOS, or other possible operating systems; this embodiment does not impose specific limitations.

[0235] The food ingredient recognition device provided in this embodiment can achieve... Figures 1 to 4The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0236] Figure 6 This is a schematic diagram of the structure of the food ingredient recognition system provided in this application embodiment, such as... Figure 6 As shown, the food identification system 60 for hands-on food placement includes: a cooling device 601, a first camera 602, and an electronic device 603, wherein,

[0237] The refrigeration device 601 has multiple non-transparent shelves inside its casing.

[0238] The first camera 602 is installed in the middle of the top of the inner liner of the refrigeration device 601 and tilted towards the shelf;

[0239] The electronic device 603 includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for recognizing food placed by both hands as described in the above method embodiments.

[0240] In some embodiments, such as Figure 7 As shown, this application embodiment also provides an electronic device 603, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, it implements the various processes of the above-described method embodiment for identifying food placed by both hands, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0241] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0242] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described method embodiment for identifying food ingredients placed by both hands, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0243] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0244] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-mentioned method for recognizing food ingredients placed by both hands.

[0245] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0246] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described method embodiment for identifying food placed by both hands, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0247] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0248] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0249] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0250] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0251] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0252] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for recognizing food ingredients placed by both hands, characterized in that, include: Acquire the first image of multiple frames; The hands and food ingredients are detected and identified in the multiple frames of the first image, and the detection and identification results of the hands and food ingredients in each first image are obtained. Based on the hand and food detection and recognition results corresponding to each first image in the multi-frame first image, the user's hand movements are tracked to determine whether there is at least one hand movement for putting food in the food. If at least one of the user's hand movements involves placing food, the target shelf area for placing food is determined based on the target image of the bounding box of the hand holding the food in the multi-frame first image, which includes the hand and food detection and recognition results.

2. The method for recognizing food ingredients placed by both hands according to claim 1, characterized in that, The first multi-frame image is captured by the first camera, or obtained by processing the video captured by the first camera. The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and tilted towards the shelf. The inner liner is provided with multiple non-transparent shelves.

3. The method for recognizing food ingredients placed by both hands according to claim 1 or 2, characterized in that, The hand and food detection and identification results include at least one of the following: The bounding box of holding ingredients; The bounding box of the hand; null.

4. The method for recognizing food ingredients placed by both hands according to claim 3, characterized in that, The step of tracking the user's hand movements based on the hand and food detection and recognition results corresponding to each of the multiple first images, and determining whether at least one hand is used to place food, includes: If the hand and food detection and recognition results corresponding to the multi-frame first image meet the first preset condition, then it is determined that at least one of the user's hand movements is putting food into the container. The first preset conditions include: Among the hand and food detection and recognition results corresponding to the multiple first images, the hand and food detection and recognition results of N consecutive images are one of the following: Empty; the bounding box of the hand holding the ingredients is empty. Empty, the bounding box of the first hand holding the food, the bounding box of the second hand holding the food, and the absolute value of the difference between the Y value of the bounding box of the first hand holding the food and the Y value of the bounding box of the second hand holding the food is less than or less than or equal to the first value; Empty, the bounding box of the hand holding the ingredients, the bounding box of the hand.

5. The method for recognizing food ingredients placed by both hands according to claim 4, the method further includes: If the hand and food detection and recognition results corresponding to the first multi-frame first image do not meet the first preset condition, then it is determined that there is no action of at least one hand in the user's hand movements to put in food, and the process ends.

6. The method for recognizing food ingredients placed with both hands according to claim 1, characterized in that, The detection and recognition results of hands and food in the multi-frame first images include the target image of the bounding box of the hand holding the food, and determining the target shelf area where the hands place the food, including: Acquire the target image in the multi-frame first image, wherein the target image is the image of the last frame of the hand and food detection and recognition results in the multi-frame first image before the ID of the same hand disappears, including the bounding box of the hand holding the food; An image segmentation algorithm is used to segment the shelf area in the target image to obtain the shelf area segmentation result corresponding to the target image; Based on the bounding box information of the hands holding food in the target image and the segmentation result of the shelf area corresponding to the target image, the target shelf area for placing the hands into the food is determined.

7. The method for recognizing food ingredients placed with both hands according to claim 6, characterized in that, The step of determining the target shelf area for placing the food into the hands, based on the bounding box information of the hands holding the food in the target image and the shelf area segmentation result corresponding to the target image, includes: If the hand and food detection and recognition results corresponding to the target image include two bounding boxes of hands holding food, the target shelf area for placing the food is determined based on the Y value of each bounding box of the hands holding food and the shelf area segmentation result corresponding to the target image. If the detection and recognition results of the hand and food corresponding to the target image include a bounding box of a hand with a first ID and a bounding box of a hand holding food with a second ID, the target shelf area where the hand with the second ID puts the food is determined based on the Y value of the bounding box of the hand holding food with the second ID and the shelf area segmentation result corresponding to the target image.

8. The method for recognizing food ingredients placed by both hands according to claim 7, characterized in that, The method further includes: Control the reminder component corresponding to the target shelf area to output a prompt message.

9. A device for identifying food ingredients placed by both hands, characterized in that, include: An image acquisition unit is used to acquire multiple frames of the first image; The detection and recognition unit is used to detect and recognize hands and food in the multiple frames of first images respectively, and to obtain the detection and recognition results of hands and food in each first image in the multiple frames of first images; The first determining unit is used to track the user's hand movements based on the hand and food detection and recognition results corresponding to each first image in the multiple first images, and to determine whether there is at least one hand movement for putting food in the user's hand movements. The second determining unit is used to determine the target shelf area for placing food by both hands, based on the target image of the bounding box of the hand holding the food in the multi-frame first image, when at least one of the user's hand movements is to put food in.

10. A recognition system for food ingredients placed by both hands, characterized in that, include: The cooling unit, the first camera, and electronic equipment, among which, The refrigeration unit has multiple non-transparent shelves inside its casing; The first camera is installed in the middle of the top of the inner liner of the refrigeration unit and is tilted towards the shelf. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the identification method for placing food ingredients by both hands as described in any one of claims 1-8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for recognizing food ingredients placed by both hands as described in any one of claims 1-8.

Images