Method for identifying items in refrigerator, refrigerator, and computer storage medium

Abstract

The present invention provides a method for identifying items in a refrigerator, a refrigerator, and a computer storage medium. The method includes the following steps: obtaining multiple images of the refrigerator storage space taken at the same point and selecting one of the images as a reference image; detecting, identifying, and labeling item information within the reference image; comparing the remaining images with the reference image and, based on the degree of repetition between the two, segmenting the remaining image content into similar areas and different areas; using the corresponding item information in the reference image to identify the similar areas, extracting images of the different areas for secondary detection and identification, and performing secondary labeling. By comparing multiple images, segmenting the different areas in the images based on the degree of repetition, and performing secondary image extraction and identification on the different areas, the accuracy of identifying items in the images is improved and the occurrence of missed identification is reduced.

Description

Technical Field

[0001] The present invention relates to the field of refrigeration devices, and in particular to a method for identifying items in a refrigerator, a refrigerator, and a computer storage medium. Background Art

[0002] With the development of intelligent home appliances, identifying internal items has become a necessary function for refrigerators. To achieve this, one or more cameras are installed in the refrigerator's internal storage space to capture the items and process the captured images to generate recognition results.

[0003] However, during the image detection and recognition process, due to various reasons such as lighting, the camera itself, and placed objects, the image quality may not be good enough, or the image detection algorithm itself may not have a high enough generalization ability, resulting in low recognition accuracy. Missed recognition often occurs, especially when the position of the object in the current and subsequent images has not changed. When the object detected in one image is not detected in the other, it seriously affects the user experience. Summary of the Invention

[0004] The object of the present invention is to provide a.

[0005] The present invention provides a method for identifying items in a refrigerator, which is characterized by comprising the steps of:

[0006] Acquire multiple images of the refrigerator storage space captured at the same point, and select one of the images as a reference image;

[0007] Detecting, identifying and labeling object information within the reference image;

[0008] comparing the remaining images with the reference image, and dividing the remaining image contents into similar areas and different areas based on the degree of repetition between the two;

[0009] The corresponding item information in the reference image is used to mark the similar area, and the image of the different area is extracted for secondary detection and recognition, and secondary marking is performed.

[0010] As a further improvement of the present invention, “obtaining multiple refrigerator storage space images captured at the same point” specifically includes:

[0011] After each detection meets the shooting requirements, take pictures respectively, or

[0012] Multiple images are taken at fixed time intervals, or

[0013] Video images are captured over a period of time, and the video is intercepted at fixed frame intervals to obtain multiple images.

[0014] As a further improvement of the present invention, “detecting and identifying object information in the reference image” specifically includes:

[0015] Based on the detection algorithm model, different objects in the reference image are obtained and their target detection frames are identified;

[0016] Based on the classification network model, the objects in the target detection frame are classified and identified, their information is obtained, and the objects are marked.

[0017] As a further improvement of the present invention, “comparing the remaining images with the reference image and dividing the remaining image contents into similar areas and different areas based on the repetition between the two” specifically includes:

[0018] Obtaining target detection frames of different objects in the remaining images;

[0019] translating the target detection frame of the reference image to the remaining images, calculating the intersection-over-union ratio between the target detection frames, eliminating the target detection frames whose intersection-over-union ratio with any target detection frame is greater than a preset threshold, and marking the remaining target detection frames as secondary target detection frames;

[0020] The area within the secondary target detection frame is identified as the difference area, and the rest of the area is identified as the similar area.

[0021] As a further improvement of the present invention, “the similar region uses the object information in the reference image and extracts the image of the difference region for secondary detection and identification” specifically includes:

[0022] directly marking the item identification information corresponding to the similar area in the reference image at the same position in the remaining images;

[0023] Extracting the difference area from the remaining images to obtain a secondary detection image;

[0024] Based on the detection algorithm model and the classification network model, the objects in the secondary detection image are classified and identified to obtain their information.

[0025] As a further improvement of the present invention, “performing secondary identification” specifically includes:

[0026] After the object information in the secondary detection image is detected and identified, it is marked on the corresponding extracted part of the original image;

[0027] When no object is detected in the secondary detection image, the original image is not processed.

[0028] As a further improvement of the present invention, “performing secondary identification” also includes:

[0029] When the object information in the secondary detection image is detected and identified, and it is detected that the object does not exist in the reference image, the secondary target detection frame corresponding to the object is translated into the reference image, and the corresponding area is extracted to obtain a tertiary detection image;

[0030] Based on the detection algorithm model and the classification network model, the objects in the secondary detection image are classified and identified to obtain their information.

[0031] As a further improvement of the present invention, the detection algorithm model is the YOLOv4 target detection algorithm, and the classification network model is based on the ResNet18 deep convolutional neural network.

[0032] The present invention also provides a refrigerator, comprising: a camera, a memory, and a processor, wherein the camera is configured to capture an image of a storage space in the refrigerator;

[0033] The memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the above-mentioned method for identifying items in the refrigerator are implemented.

[0034] As a further improvement of the present invention, the refrigerator includes a plurality of bottle holders, which are arranged in sequence from top to bottom on the refrigerator door. The camera is arranged at the bottom of the bottle holders, vertically downward, for capturing images in the bottle holder space below them.

[0035] The beneficial effects of the present invention are: by comparing multiple images, different areas are divided in the image according to the difference in repetition degree, and the different areas are subjected to secondary cutout recognition, the accuracy of identifying objects in the image is improved and the occurrence of missed recognition is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 1 is a schematic diagram of the steps of a method for identifying items in a refrigerator in one embodiment of the present invention.

[0037] Figures 2 to 7 1 is a schematic diagram illustrating the steps of a method for identifying items in a refrigerator in one embodiment of the present invention. DETAILED DESCRIPTION

[0038] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0039] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0040] For ease of explanation, this document uses terms that indicate spatial relative positions, such as "upper," "lower," "rear," "front," etc., to describe the relationship of one unit or feature shown in the drawings relative to another unit or feature. Terms that indicate spatial relative positions may include different orientations of the device during use or operation other than the orientation shown in the drawings. For example, if the device in the drawings is turned over, units described as being "below" or "above" other units or features will be located "below" or "above" the other units or features. Therefore, the exemplary term "below" can encompass both below and above spatial orientations.

[0041] like Figure 1 As shown, the present invention provides a method for identifying items in a refrigerator, which is used to calibrate and correct the recognition results in the captured image, comprising the steps of:

[0042] S1: Obtain multiple refrigerator storage space images taken at the same point and select one of the images as a reference image.

[0043] Shooting at the same point here means that a camera is set at a fixed point and kept relatively stable between it and the viewing area to obtain multiple images, thereby ensuring to the greatest extent possible that the position of the object in each image remains the same, facilitating subsequent detection and identification.

[0044] For example, in this embodiment, the camera is located at the bottom of a refrigerator bottle to capture information about the food in the bottle holder below. In other embodiments of the present invention, the camera can also be located in other storage spaces within the refrigerator, such as the refrigerator or freezer, and its location can be adjusted based on the structure of the refrigerator or freezer.

[0045] Specifically, each time it is detected that the shooting requirement is met, an image is shot respectively, and the image obtained by the last shooting is used as a reference image for correcting the image after each shooting.

[0046] In other embodiments of the present invention, multiple images can be captured in sequence at fixed time intervals. After detecting that the shooting requirements are met, such as detecting that the refrigerator door is closed or detecting a shooting command manually input by the user, the camera is started to shoot; or, video images over a period of time can be captured, and the video can be intercepted at fixed frame intervals to obtain multiple images. The above two shooting methods are applied to scenarios where multiple images are captured at one time and the multiple images are corrected at the same time.

[0047] For example, Figure 2 and Figure 3 The following are schematic diagrams of the first and second images captured in this embodiment. The first image contains items 1, 2, 3, 4, and 5. In the second image, item 3 is displaced due to the impact of the refrigerator door closing. Item 4 is removed by the user after the camera starts recording.

[0048] S2: Detect, identify and label the object information in the reference image.

[0049] Specifically, based on the detection algorithm model, different objects in the reference image are obtained and their target detection frames are identified. For example, objects can be detected using common object detection algorithms such as YOLOv4. The target detection frame is typically a rectangular or quadrilateral detection frame that encloses a single object.

[0050] Based on the classification network model, the items in the target detection frame are classified and identified, their information is obtained, and the items are labeled. For example, the ResNet18 deep convolutional neural network can be trained and the items that may be stored in the refrigerator can be classified and identified based on the trained ResNet18 network model.

[0051] The first to the last of the multiple images taken continuously can be selected as the reference images to make the subsequent detection and judgment logic more consistent.

[0052] For example, Figure 4 As shown, in this embodiment, the first image is used as the reference image to detect and identify the items therein, wherein items 1 to 4 are detected and identified, and are marked with target detection frame a and item information, while item 5 is missed.

[0053] S3: Compare the remaining images with the reference image, and segment the remaining image contents into similar areas and different areas based on the degree of repetition between the two.

[0054] Specifically, step S3 includes the following steps:

[0055] S31: Obtain target detection frames of different objects in the remaining images.

[0056] When there are multiple remaining images, they can be detected sequentially in the order of shooting or in reverse order.

[0057] For example, Figure 5 As shown, in this embodiment, the target detection frame in the second image is obtained, wherein the object 2 is missed.

[0058] S32: translating the target detection frame of the reference image to the remaining images, calculating the intersection-over-union ratio between the target detection frames, eliminating the target detection frames whose intersection-over-union ratio with any target detection frame is greater than a preset threshold, and marking the remaining target detection frames as secondary target detection frames.

[0059] The intersection-over-union (IoU) ratio refers to the area ratio of the overlapping part of two target detection frames to the combined part. It can be used to determine the degree of overlap between the two areas. When the IoU ratio between the two target detection frames is greater than the preset threshold, since the multiple images are taken continuously, it can be determined that the object in the corresponding area has not changed.

[0060] When there are multiple remaining images, their target detection frames can be compared with the reference image in the order or in reverse order of shooting. Alternatively, the reference image can be compared with a set of images for calibration before comparing the remaining images with the reference image.

[0061] For example, Figure 6 As shown in the figure, the detection frame in the first image is translated to the second image, where the target detection frame of item 1 overlaps, the intersection-over-union ratio of the target detection frame of item 3 is less than the threshold, and the target detection frames of items 2, 4, and 5 have no corresponding target detection frames. The target detection frames corresponding to items 3, 4, and 5 are identified as secondary detection frames.

[0062] S33: Identify the area within the secondary target detection frame as a difference area, and identify the remaining areas as similar areas.

[0063] A similar area means that the objects and their placement in the area in the two images can be judged to have not changed significantly. A different area means that there is a deviation in the object recognition process in the area in the two images, which may be a missed recognition or a different placement of the objects.

[0064] S4: Using the corresponding object information in the reference image to mark the similar area, extracting the image of the different area for secondary detection and recognition, and performing secondary marking.

[0065] Specifically, for similar areas, it is considered that no significant changes have occurred, and the object identification information corresponding to the similar areas in the reference image is directly marked at the same position in the remaining images.

[0066] The different areas in the rest of the image are cut out to obtain a secondary detection image. Cutting out the different areas can reduce the interference of other invalid information in the image on the object to be identified, thereby improving the recognition accuracy and reducing the occurrence of missed recognition.

[0067] Based on the detection algorithm model and classification network model, the objects in the secondary detection image are classified and identified to obtain their information.

[0068] For different secondary recognition results, when the object information in the secondary detection image is detected and recognized, it is marked in the corresponding extracted part of the original image.

[0069] When no object is detected in the secondary detection image, the original image is not processed.

[0070] For example, Figure 7 As shown, after the second recognition of the difference area, items 2, 3 and 5 are detected and identified and marked, while item 4 is not detected. It is confirmed that item 4 in the first image does not exist in the second image and is not marked.

[0071] Furthermore, in applications where multiple images are captured in a single shot, the baseline image must be detected and corrected. When an object in the secondary detection image is detected and not present in the baseline image, the secondary object detection frame corresponding to the object is translated into the baseline image, and the corresponding area is extracted to obtain the tertiary detection image. Based on the detection algorithm model and classification network model, the objects in the tertiary detection images are classified and identified to obtain their information.

[0072] For example, object 5 is detected in the second image, but not in the first image. Therefore, the target detection frame corresponding to object 5 in the second image is translated into the first image, and detection and recognition are performed after three detection images are obtained.

[0073] The present invention also provides a refrigerator, comprising: a camera, a memory, and a processor, wherein the camera is configured to capture images of storage space in the refrigerator.

[0074] The memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the above-mentioned method for identifying items in the refrigerator are implemented.

[0075] The refrigerator includes a plurality of bottle holders, which are arranged in sequence from top to bottom on the refrigerator door. A camera is arranged at the bottom of the bottle holders, vertically downward, for capturing images in the space below the bottle holders.

[0076] In summary, the present invention compares multiple images, divides different areas in the image according to the difference in repetition degree, and performs secondary cutout recognition on the different areas, thereby improving the accuracy of identifying objects in the image and reducing the occurrence of missed recognition.

[0077] It should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation method can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0078] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying items in a refrigerator, characterized in that: Including steps: Acquire multiple images of the refrigerator storage space captured at the same point, and select one of the images as a reference image; Detecting, identifying and labeling object information within the reference image; comparing the remaining images with the reference image, and dividing the remaining image contents into similar areas and different areas based on the degree of repetition between the two; Using the corresponding item information in the reference image to mark the similar area, extracting the image of the different area for secondary detection and recognition, and performing secondary marking; Among them, based on the secondary identification result, when the object information in the secondary detection image is detected and identified, it is marked in the corresponding extracted part of the remaining images; when the object is not detected in the secondary detection image, the remaining images are not processed; the secondary detection image is obtained based on the extraction of the difference area in the remaining images.

2. The method for identifying items in a refrigerator according to claim 1, wherein: "Acquiring multiple images of the refrigerator storage space taken at the same point" specifically includes: After each detection meets the shooting requirements, take pictures respectively, or Multiple images are taken at fixed time intervals, or Video images are captured over a period of time, and the video is intercepted at fixed frame intervals to obtain multiple images.

3. The method for identifying items in a refrigerator according to claim 1, wherein: “Detecting and identifying object information within the reference image” specifically includes: Based on the detection algorithm model, different objects in the reference image are obtained and their target detection frames are identified; Based on the classification network model, the objects in the target detection frame are classified and identified, their information is obtained, and the objects are marked.

4. The method for identifying items in a refrigerator according to claim 3, wherein: “Comparing the remaining images with the reference image and segmenting the remaining image contents into similar areas and different areas based on the degree of repetition between the two” specifically includes: Obtaining first target detection frames of different objects in the remaining images; translating the target detection frame of the reference image to the remaining images, calculating the intersection-over-union ratio between the target detection frame and the first target detection frame, eliminating the first target detection frame whose intersection-over-union ratio with any target detection frame is greater than a preset threshold, and marking the remaining first target detection frames as secondary target detection frames; The area within the secondary target detection frame is identified as the difference area, and the rest of the area is identified as the similar area.

5. The method for identifying items in a refrigerator according to claim 4, wherein: “Using the object information in the reference image in the similar area and extracting the image of the difference area for secondary detection and identification” specifically includes: directly marking the item identification information corresponding to the similar area in the reference image at the same position in the remaining images; Based on the detection algorithm model and the classification network model, the objects in the secondary detection image are classified and identified to obtain their information.

6. The method for identifying items in a refrigerator according to claim 1, wherein: "Secondary identification" also includes: When the object information in the secondary detection image is detected and identified, and it is detected that the object does not exist in the reference image, the secondary target detection frame corresponding to the object is translated into the reference image, and the corresponding area is extracted to obtain a tertiary detection image; Based on the detection algorithm model and the classification network model, the objects in the secondary detection image are classified and identified to obtain their information.

7. The method for identifying items in a refrigerator according to claim 3, wherein: The detection algorithm model is the YOLOv4 target detection algorithm, and the classification network model is based on the ResNet18 deep convolutional neural network.

8. A refrigerator comprising: A camera, a memory and a processor, characterized in that The camera is configured to capture images of the storage space in the refrigerator; The memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the method for identifying items in the refrigerator according to any one of claims 1 to 7 are implemented.

9. The refrigerator according to claim 8, characterized in that The refrigerator includes a plurality of bottle holders, which are arranged in sequence from top to bottom on the refrigerator door. The camera is arranged at the bottom of the bottle holders, vertically downward, and is used to capture images in the bottle holder space below them.

Images