Article information recognition method, refrigeration device, and computer storage medium
By processing images and recognizing behaviors of food items inside refrigerator drawers, the problem of cameras being unable to identify food items inside drawers has been solved, enabling accurate tracking and data recording of food storage and retrieval, and improving the refrigerator's intelligent recognition capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QINDAO HAIER REFRIGERATOR CO LTD
- Filing Date
- 2023-03-30
- Publication Date
- 2026-07-10
AI Technical Summary
Existing refrigerator cameras, due to structural limitations and user habits, cannot fully identify all food items in drawers when food is stored inside, leading to data discrepancies.
By processing continuous video frames, the system identifies the outside of the drawer, the visible area, and the occluded area. Using target detection and recognition algorithms, it tracks hand and food features to determine the food storage and retrieval behavior, thus achieving accurate identification of the food inside the drawer.
Even when the space inside the drawer is partially obscured, it can still effectively record food data, providing more convenient user services and improving the accuracy and completeness of food identification.
Smart Images
Figure CN116416556B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of refrigeration devices, and more particularly to a method for identifying item information, a refrigeration device, and a computer storage medium. Background Technology
[0002] With the development of smart appliances, the ability to identify items inside has become an essential function of smart refrigerators. Typically, to identify items stored inside the refrigerator, one or more cameras are installed to capture images of the items. These images are then processed and sent to the user's device for viewing.
[0003] However, when adding food to the drawers in the refrigerator's camera-controlled storage area, the drawers may not be fully opened due to structural limitations and user experience. This creates blind spots inside the drawers. For example, if the drawer has an opaque shelf, the camera cannot capture the contents of the drawer when it's partially open or not fully open. Conversely, if the shelf is transparent, other food items will also obstruct the view of the drawer's contents. When users store food in these blind spots, conventional image analysis cannot identify all the food in the drawer, leading to inaccuracies in the refrigerator control center's data on food storage. Summary of the Invention
[0004] The purpose of this invention is to provide a method for identifying item information, a refrigeration device, and a computer storage medium.
[0005] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides a method for identifying item information, comprising the following steps:
[0006] Image processing is performed on each video frame image captured in succession. Specifically, a bounding box composed of intersecting contour lines is identified in each video frame image and defined as a first pixel region, a second pixel region, and a third pixel region. The first pixel region is the outer area of the drawer, the second pixel region is the visible area inside the drawer, and the third pixel region is the occluded area inside the drawer. Multiple detection boxes are segmented in each video frame image, each detection box corresponding to a feature item, and the feature item within the detection box is identified. The feature item includes food features and hand features.
[0007] When the detection box for the hand feature overlaps with the detection box for the food feature, the system enters the access mode.
[0008] In the access mode, the detection boxes of the food features that overlap with the detection boxes of the hand features are continuously tracked;
[0009] When the detection box of the food feature that overlaps with the detection box of the hand feature disappears in the video frame image, then the video frame image before the detection box of the food feature disappears is obtained.
[0010] If the center point of the detection box of the food ingredient feature is located in the first pixel region in a video frame image before the detection box of the food ingredient feature disappears, then the food ingredient is determined to be removed.
[0011] If the center point of the detection box of the food ingredient feature is located in the second pixel region in the video frame image before the detection box of the food ingredient feature disappears, then the food ingredient is determined to be stored.
[0012] As a further improvement of one embodiment of the present invention, when the drawer is opened, a start shooting command is triggered; when the drawer is closed, a stop shooting command is triggered.
[0013] As a further improvement to one embodiment of the present invention, when the detection frame of the hand feature overlaps with the detection frame of the food feature, the system enters the access mode, specifically including:
[0014] Establish a coordinate system in each video frame image and obtain the coordinate range of each feature item detection box;
[0015] When the coordinate range of the detection box for the hand feature partially overlaps with the coordinate range of the detection box for the food feature, the system enters access mode.
[0016] As a further improvement to one embodiment of the present invention, when the detection frame of the hand feature overlaps with the detection frame of the food feature, the system enters the access mode, specifically further including:
[0017] If the detection boxes of the hand features and the detection boxes of the food features overlap in at least 5-10 consecutive video frames, then this behavior is determined to be holding food.
[0018] When the aforementioned action of holding food ingredients occurs, the system enters the storage and retrieval mode.
[0019] As a further improvement to one embodiment of the present invention, the step of continuously tracking the detection frame of the food feature that overlaps with the detection frame of the hand feature in the access mode further includes:
[0020] In the access mode, when the detection boxes of the overlapping hand features and the food feature are separated, and the center point of the detection box of the food feature is located in the second pixel area in each video frame image after the separation of the two detection boxes, it is determined that the food is stored.
[0021] As a further improvement to one embodiment of the present invention, the method further includes:
[0022] When the detection box for the hand feature disappears from the video frame image, the system closes the access mode.
[0023] As a further improvement to one embodiment of the present invention, the method further includes:
[0024] The information on the ingredients stored in the drawer is updated, including the type and quantity of the ingredients.
[0025] As a further improvement of one embodiment of the present invention, multiple detection boxes are segmented in each video frame image, each detection box corresponding to a feature item, and the feature item within the detection box is identified. The feature item includes food features and hand features, specifically including:
[0026] Using an object detection algorithm, the detection box corresponding to each feature item in the video frame image is marked;
[0027] The target recognition algorithm is used to identify the feature items within each detection box.
[0028] This invention provides a refrigeration device, comprising: a cabinet, a shelf, a drawer, a camera, a memory, and a processor, wherein...
[0029] The drawer and the shelf are disposed inside the box. The drawer has an inner space. The shelf is disposed above the drawer and makes the inner space of the drawer a closed inner space. When the drawer slides outward toward the box, part of the inner space of the drawer is exposed to the outside.
[0030] The shelf is an opaque shelf;
[0031] The camera is installed inside the box, vertically downward, and is used to capture images including the space inside the drawer.
[0032] The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps of the item information identification method as described in any of the above embodiments.
[0033] The present invention provides a computer storage medium storing a computer program, wherein the computer program, when executed, causes the device containing the computer storage medium to perform the steps of the item information identification method as described in any of the above embodiments.
[0034] The advantages of this invention are: by using a camera to capture the process of a user adding or taking food into a drawer, even if the space inside the drawer is partially obscured and the entire space inside the drawer cannot be captured, the image processing method can track the direction of the user's hand holding the food and determine whether the food is being put into the drawer or taken out of the drawer, thus more effectively recording the data information of the food placed in the drawer and providing users with a more convenient service. Attached Figure Description
[0035] Figure 1 This is a schematic flowchart of an item information identification method according to one embodiment of the present invention;
[0036] Figure 2 This is a schematic diagram of the drawer when it is closed according to one embodiment of the present invention (the area outside the drawer and the area inside the drawer that are covered).
[0037] Figure 3 This is a schematic diagram of a video frame image in access mode according to an embodiment of the present invention;
[0038] Figure 4 This is a simplified schematic diagram of a video frame image before the food is removed in the storage and retrieval mode according to one embodiment of the present invention.
[0039] Figure 5 This is a simplified schematic diagram of a video frame image before food ingredients are stored in the storage mode according to one embodiment of the present invention. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0041] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0042] For ease of explanation, this document uses terms indicating relative spatial position, such as "above," "below," "behind," and "front," to describe the relationship of one unit or feature shown in the accompanying drawings relative to another unit or feature. Terms indicating relative spatial position can include different orientations of the device during use or operation besides those shown in the figures. For example, if the device in the figures is flipped, a unit described as being "below" or "above" other units or features will be located "below" or "above" other units or features. Therefore, the exemplary term "below" can encompass both "below" and "above" spatial orientations.
[0043] like Figure 1 The diagram illustrates an item information recognition method provided in this embodiment, particularly a method for recognizing items inside a partially obscured drawer, comprising the following steps:
[0044] S1: Perform image processing on each video frame image captured in succession. In each video frame image, identify the outline bounding boxes formed by intersecting contour lines, and define them as the first pixel region, the second pixel region, and the third pixel region. The first pixel region is the outer region of the drawer, the second pixel region is the visible region inside the drawer, and the third pixel region is the occluded region inside the drawer. Segment multiple detection boxes in each video frame image. Each detection box corresponds to a feature item, and identify the feature item within the detection box. The feature items include food features and hand features.
[0045] S2: When the detection box for hand features overlaps with the detection box for food features, the system enters access mode;
[0046] S3: In access mode, continuously track the detection boxes of food features that overlap with the detection boxes of hand features;
[0047] S4: When the detection box of the food feature that overlaps with the detection box of the hand feature disappears in the video frame image, then obtain the video frame image before the detection box of the food feature disappears.
[0048] S51: If the center point of the detection box of the food feature is located in the first pixel area in the video frame image before the food feature disappears, then the food is determined to be removed.
[0049] S52: If the center point of the detection box of the food feature is located in the second pixel region in the video frame image before the food feature disappears, then the food is determined to be stored.
[0050] Specifically, when the drawer is opened, a shooting command is triggered; when the drawer is closed, a shooting command is triggered.
[0051] In step S1, the bounding boxes formed by intersecting contour lines are identified in each video frame image and defined as the first pixel region, the second pixel region, and the third pixel region, specifically including:
[0052] When a drawer is opened, due to the differences in material, color, and brightness between the outer area of the drawer, the visible area inside the drawer, and the obscured area inside the drawer (i.e., the invisible area inside the drawer), the outer area of the drawer, the visible area inside the drawer, and the obscured area inside the drawer captured by the camera usually have obvious pixel differences in each video frame image, and can be detected and identified as having obvious edge contours.
[0053] In one embodiment of the present invention, an edge detection algorithm is used to identify the contour lines intersecting the outer region of the drawer, the visible region of the drawer's interior space, and the occluded region of the drawer's interior space in each video frame image. These contour lines form three bounding boxes. The outer region of the drawer is defined as the first pixel region A, the visible region of the drawer's interior space is defined as the second pixel region B, and the occluded region of the drawer's interior space is defined as the third pixel region C. Figures 2-5 .
[0054] Here, edge detection algorithms such as Roberts, Sobel, Prewitt, Canny, and Log can be used. The specific algorithm content is existing technology and will not be elaborated on here.
[0055] In step S1, multiple detection boxes are segmented in each video frame image, each detection box corresponding to a feature item, and the feature item within the detection box is identified. The feature item includes food features and hand features, specifically including:
[0056] Using object detection algorithms, the detection box corresponding to each feature item in the video frame image is marked;
[0057] The target recognition algorithm is used to identify the feature items within each detection box.
[0058] Specifically, the YOLO v4 object detection algorithm is used to detect feature objects in each video frame image and generate a corresponding detection box at each feature object.
[0059] Specifically, the ResNet object recognition algorithm is used to identify the feature items within each detection box. These feature items include food features and hand features. For food feature detection boxes, the ResNet object recognition algorithm can identify the type of food based on the top image of the food, its shape, color, etc.
[0060] In other embodiments of the present invention, other existing target detection algorithms and target recognition algorithms may also be used to generate object detection boxes for objects in bird's-eye view images and identify their object information.
[0061] In step S2, when the detection bounding box for hand features overlaps with the detection bounding box for food features, the system enters access mode, specifically including:
[0062] Establish a coordinate system in each video frame image and obtain the coordinate range of each feature object detection box;
[0063] When the coordinate range of the detection box for hand features partially overlaps with the coordinate range of the detection box for food features, the system enters access mode.
[0064] Specifically, a coordinate system is established with one vertex in the video frame image as the origin and the two sides connecting that vertex as the x-axis and y-axis.
[0065] For example, with Figure 3 Point O is the origin. The x-axis is a side parallel to the first pixel region A, and the positive x-axis direction is the direction towards the edge of the video frame image. The y-axis is a side perpendicular to the first pixel region A, and the positive y-axis direction is the direction towards the third pixel region C.
[0066] Based on this, it is convenient to obtain the coordinate range of the detection box for food features and the coordinate range of the detection box for hand features. When the coordinate range of the detection box for hand features partially overlaps with the coordinate range of the detection box for food features, the system enters the access mode.
[0067] Furthermore, when the detection boxes for hand features and food features overlap in at least 5-10 consecutive video frames, the behavior is determined to be holding food; when the behavior of holding food occurs, the system enters the access mode.
[0068] For example, such as Figures 3-5 As shown, when the coordinate range of the detection box 1 for hand features and the coordinate range of the detection box 2 for food features overlap in the video frame image, and the detection and recognition results are the same in at least 5-10 consecutive video frame images, then the behavior is judged to be holding food, and the system enters the storage and retrieval mode.
[0069] In access mode, the detection boxes 2 of food features that overlap with the detection box 1 of the hand feature are continuously tracked. When the detection box 2 of the food feature that overlaps with the detection box 1 of the hand feature disappears from the video frame image, the video frame image before the detection box 2 of the food feature disappears is acquired.
[0070] There are two possibilities: the food item may have been taken out by the user, or it may have been placed in a concealed area inside the drawer, out of the camera's view. Therefore, the judgment is made based on the video frame image before the detection box 2 of the food item's features disappears.
[0071] For example, such as Figure 4 As shown, if the center point of the detection box 2 of the food feature is located in the first pixel region A in a video frame image before the food feature detection box 2 disappears, then the food is determined to be removed.
[0072] For example, such as Figure 5 As shown, if the center point of the detection box 2 of the food feature is located in the second pixel region B in a video frame image before the food feature detection box 2 disappears, then the food is determined to be stored.
[0073] In another embodiment of the present invention, the determination can also be made based on the coordinate range in which the center point of the detection box 2 of the food feature falls in a video frame image before the food feature disappears. For example, based on the coordinate system established in each video frame image, the coordinate ranges of the drawer outer area, the visible area inside the drawer, and the obscured area inside the drawer are marked. The determination of whether the food is taken out or stored is based on which coordinate range the center point of the detection box 2 of the food feature falls in a video frame image before the food feature disappears.
[0074] Of course, the food is not necessarily placed in the obscured area of the drawer (i.e., disappearing from the video frame image). The food can also be placed in the visible area of the drawer, i.e., in the second pixel area B.
[0075] In access mode, when the detection box 1 of overlapping hand features and the detection box 2 of food features are separated, and the center point of the detection box 2 of food features is located in the second pixel area in each video frame image after the separation of the two detection boxes, then the food is determined to be stored.
[0076] Specifically, when a user places food items in the visible area of the drawer, upon entering the storage / retrieval mode, if the overlapping area between the coordinate ranges of the detection box 1 (for hand features) and the detection box 2 (for food features) decreases until they no longer overlap, a video frame image after the two are separated is acquired. It is then determined whether the center point of the detection box 2 (for food features) is located within the second pixel region B. If the center point of the detection box 2 (for food features) is located within the second pixel region B, then the food item is determined to be stored.
[0077] Furthermore, when the detection box 1 for the detected hand features disappears from the video frame image, the system closes the access mode.
[0078] Furthermore, based on a series of video frame images acquired by the camera, the movement direction of the detection boxes for hand features and food features when holding food in the storage mode is tracked, and the information of the food stored in the drawer is updated. The food information includes the type and quantity of food.
[0079] The present invention provides a refrigeration device, comprising: a cabinet, a shelf, a drawer, a camera, a memory, and a processor.
[0080] In a specific embodiment of the present invention, the refrigeration device is a smart refrigerator. The type and quantity of food placed in the refrigerator's designated area can be interacted with by the user through voice interaction or display on the refrigerator's large screen.
[0081] The drawers and shelves are located inside the box. The drawers have an inner space, and the shelves are located above the drawers, making the inner space of the drawers a closed inner space. When the drawers slide outwards from the box, part of the inner space of the drawers is exposed to the outside.
[0082] Specifically, the shelves are opaque.
[0083] The camera is installed inside the cabinet, vertically downwards, to capture images including the interior space of the drawers. When the drawer begins to slide outwards, exposing the interior space, the camera starts capturing images; when the drawer is closed, the camera stops capturing images.
[0084] The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps of the item information identification method in any of the above embodiments.
[0085] The present invention also provides a computer storage medium storing a computer program, wherein the computer program, when executed, causes the device on which the computer storage medium is located to perform the steps of the article information identification method in any of the above embodiments.
[0086] In summary, this invention utilizes a camera to capture the process of a user adding or removing food from a drawer. Even if the space inside the drawer is partially obscured and cannot be fully captured, image processing techniques are used to track the direction of the user's hand movement when holding the food, determining whether the food is being placed in or removed from the drawer. This more effectively records data on the food placed inside the drawer, providing users with a more convenient service.
[0087] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
[0088] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for identifying item information, characterized in that, Including the following steps: Image processing is performed on each video frame image captured in succession. Specifically, a bounding box composed of intersecting contour lines is identified in each video frame image and defined as a first pixel region, a second pixel region, and a third pixel region. The first pixel region is the outer area of the drawer, the second pixel region is the visible area inside the drawer, and the third pixel region is the occluded area inside the drawer. Multiple detection boxes are segmented in each video frame image, each detection box corresponding to a feature item, and the feature item within the detection box is identified. The feature item includes food features and hand features. When the detection box for the hand feature overlaps with the detection box for the food feature, the system enters the access mode. In the access mode, the detection boxes of the food features that overlap with the detection boxes of the hand features are continuously tracked; When the detection box of the food feature that overlaps with the detection box of the hand feature disappears in the video frame image, then the video frame image before the detection box of the food feature disappears is obtained. If the center point of the detection box of the food ingredient feature is located in the first pixel region in a video frame image before the detection box of the food ingredient feature disappears, then the food ingredient is determined to be removed. If the center point of the detection box of the food ingredient feature is located in the second pixel region in the video frame image before the detection box of the food ingredient feature disappears, then the food ingredient is determined to be stored.
2. The item information identification method according to claim 1, characterized in that, When the drawer is opened, a start shooting command is triggered; when the drawer is closed, a stop shooting command is triggered.
3. The method for identifying item information according to claim 1, characterized in that, When the detection frame of the hand feature overlaps with the detection frame of the food feature, the system enters the access mode, specifically including: Establish a coordinate system in each video frame image and obtain the coordinate range of each feature item detection box; When the coordinate range of the detection box for the hand feature partially overlaps with the coordinate range of the detection box for the food feature, the system enters access mode.
4. The article information identification method according to claim 3, characterized in that, When the detection frame of the hand feature overlaps with the detection frame of the food feature, the system enters the access mode, which further includes: If the detection boxes of the hand features and the detection boxes of the food features overlap in at least 5-10 consecutive video frames, then this behavior is determined to be holding food. When the aforementioned action of holding food ingredients occurs, the system enters the storage and retrieval mode.
5. The method for identifying item information according to claim 1, characterized in that, In the access mode, continuously tracking the detection boxes of the food features that overlap with the detection boxes of the hand features specifically includes: In the access mode, when the detection boxes of the overlapping hand features and the food features are detected to be separated, and the center point of the detection box of the food features is located in the second pixel area in each video frame image after the separation of the two detection boxes, it is determined that the food is to be stored.
6. The method for identifying item information according to claim 1, characterized in that, The method further includes: When the detection box for the hand feature disappears from the video frame image, the system closes the access mode.
7. The article information identification method according to claim 1 or 5, characterized in that, The method further includes: The information on the ingredients stored in the drawer is updated, including the type and quantity of the ingredients.
8. The method for identifying item information according to claim 1, characterized in that, The process involves segmenting multiple detection boxes from each video frame image, with each detection box corresponding to a feature item, and identifying the feature item within the detection box. The feature item includes food features and hand features, specifically including: Using an object detection algorithm, the detection box corresponding to each feature item in the video frame image is marked; The target recognition algorithm is used to identify the feature items within each detection box.
9. A refrigeration device, comprising: The enclosure, shelf, drawer, camera, memory, and processor are characterized by: The drawer and the shelf are disposed inside the box. The drawer has an inner space. The shelf is disposed above the drawer and makes the inner space of the drawer a closed inner space. When the drawer slides outward toward the box, part of the inner space of the drawer is exposed to the outside. The shelf is an opaque shelf; The camera is installed inside the box, vertically downward, and is used to capture images including the space inside the drawer. The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps of the article information identification method as described in any one of claims 1-8.
10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed, it causes the device containing the computer storage medium to perform the steps of the article information identification method according to any one of claims 1-8.