Method for identifying information of items in refrigerator, refrigerator, and computer storage medium

Abstract

The present invention discloses a method for identifying item information within a refrigerator, comprising the following steps: obtaining a bird's-eye view image of the space within a refrigerator bottle holder; identifying a contour frame composed of intersecting contour lines in the bird's-eye view image; segmenting the bird's-eye view image into multiple detection frames, each corresponding to an item; selecting a contour line and identifying the position of the contour line within the contour frame; selecting a detection line, the position of the detection line within the detection frame being opposite to the position of the contour line within the contour frame; if the relative position of the detection line to the contour line is the same as the position of the contour line within the contour frame, the corresponding item is a mirror image item; if the relative position of the detection line to the contour line is opposite to the position of the contour line within the contour frame, the corresponding item is a real item. The object information identification method proposed by the present invention can eliminate interference from mirror images and improve the accuracy of the recognition algorithm.

Description

Technical Field

[0001] The present invention relates to the field of refrigeration devices, and in particular to a method for identifying information of items in a refrigerator, a refrigerator, and a computer storage medium. Background Art

[0002] With the development of intelligent appliances, identifying internal items has become a necessary function for smart refrigerators. To identify items stored inside, refrigerators typically have one or more cameras installed inside to capture images of the items. These images are then processed and sent to the user's terminal for viewing. Placing the cameras above the items prevents obstruction and improves the accuracy of item recognition.

[0003] However, items placed inside the bottle holder will cast shadows on the bottle holder glass cabinet. The existing solution of installing the camera in the middle of the bottle holder often has mirror images of the items when collecting photos of the items, which can easily misidentify one item as two items, seriously affecting the recognition effect.

[0004] In the existing method, the wall surface of the bottle seat where the objects are placed is generally made of frosted material to eliminate the influence of the object's mirror image. However, this method not only increases the production process, but also affects the appearance of the entire machine. Summary of the Invention

[0005] The object of the present invention is to provide a method for identifying information of items in a refrigerator, a refrigerator, and a computer storage medium.

[0006] To achieve one of the above-mentioned objectives, an embodiment of the present invention provides a method for identifying information of items in a refrigerator, comprising the steps of:

[0007] Get a bird's-eye view of the space inside the refrigerator's bottle holder;

[0008] identifying a contour frame composed of intersecting contour lines in the bird's-eye view image;

[0009] Segmenting a plurality of detection frames in the bird's-eye view image, each detection frame corresponding to an object;

[0010] Selecting a contour line and identifying the position of the contour line within the contour frame;

[0011] Selecting a detection line, wherein the position of the detection line in the detection frame is opposite to the position of the contour line in the contour frame;

[0012] If the relative orientation of the detection line to the contour line is the same as the orientation of the contour line within the contour frame, the corresponding object is a mirror image object;

[0013] If the relative orientation of the detection line and the contour line is opposite to the orientation of the contour line in the contour frame, the corresponding object is a real object.

[0014] As a further improvement of the present invention, “obtaining a bird's-eye view image of the space inside the refrigerator bottle holder” specifically includes:

[0015] A camera arranged in the middle of the top of the bottle seat is controlled to capture a bird's-eye view image of the space inside the bottle seat.

[0016] As a further improvement of the present invention, “identifying a contour frame composed of intersecting contour lines in the bird's-eye view image” specifically includes:

[0017] Using an edge detection algorithm, demarcating the intersecting contour lines of the central area of ​​the bird's-eye view image;

[0018] The contour lines intersect to form a contour frame.

[0019] As a further improvement of the present invention, “segmenting a plurality of detection frames in the bird's-eye view image, each detection frame corresponding to an object” specifically includes:

[0020] Using the Yolo V3 object detection algorithm, calibrate the detection box corresponding to each object in the bird's-eye view image;

[0021] The objects in the bird's-eye view image include real objects placed in the bottle holder and mirror-image objects corresponding to the real objects.

[0022] As a further improvement of the present invention, before “selecting a detection line, the detection line is located in an orientation opposite to the orientation of the contour line in the contour frame”, the following steps are further included:

[0023] Get the detection boxes on both sides adjacent to the contour line.

[0024] As a further improvement of the present invention, the method specifically includes:

[0025] Selecting a contour line, where the contour line is located at the upper edge of the contour frame;

[0026] Selecting a detection line, where the detection line is located at the lower edge of the detection frame;

[0027] If the detection line is above the contour line, the corresponding object is a mirror image object;

[0028] If the detection line is below the contour line, the corresponding object is a real object.

[0029] As a further improvement of the present invention, the method specifically includes:

[0030] Selecting a contour line, the contour line is located at the lower edge of the contour frame;

[0031] Selecting a detection line, where the detection line is located at the upper edge of the detection frame;

[0032] If the detection line is above the contour line, the corresponding object is a real object;

[0033] If the detection line is below the contour line, the corresponding object is a mirror image object.

[0034] As a further improvement of the present invention, the method further comprises the steps of:

[0035] Excluding mirrored objects, the number of real objects in the bird's-eye view image is counted.

[0036] An embodiment of the present invention further provides a refrigerator, comprising: a bottle holder, a camera, a memory, and a processor, wherein:

[0037] The bottle seat comprises a top, a bottom and a wall of the bottle seat, and the camera is arranged in the middle of the top of the bottle seat and is arranged vertically downward;

[0038] The bottom of the bottle seat is used for placing items;

[0039] At least one wall surface of the bottle base is a glass surface;

[0040] The memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of any one of the above methods for identifying information of items in a refrigerator are implemented.

[0041] The present invention also provides a computer storage medium, which stores a computer program, and when the computer program is run, the device where the computer storage medium is located executes the steps of the method for identifying information of items in a refrigerator as described in any one of the above.

[0042] Compared with the existing technology, the present invention uses an edge detection algorithm and the Yolo V3 target detection algorithm to calibrate the contour box and the detection box corresponding to the object, and obtains the contour line and the detection line. The orientation of the detection line in the detection box is opposite to the orientation of the contour line in the contour box. By comparing whether the relative orientation of the detection line and the contour line is the same as the orientation of the contour line in the contour box, it is determined whether the object detected in the bird's-eye view image is a real object or a mirrored object. In this way, the interference of the mirrored object can be eliminated and the accuracy of the recognition algorithm can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a method for identifying information of items in a refrigerator in one embodiment of the present invention.

[0044] FIG2( a ) is a schematic diagram of capturing the space inside a bottle holder in a refrigerator according to a first embodiment of the present invention (except for the outline and the items on both sides for illustration, the rest of the contents in the diagram are omitted).

[0045] FIG2( b ) is a flow chart of the method for identifying information of items in a refrigerator according to the first embodiment of the present invention.

[0046] FIG3( a ) is a schematic diagram of capturing the space inside a bottle holder in a refrigerator according to a second embodiment of the present invention (except for the outline used for explanation and the items on both sides, the rest of the content in the figure is omitted).

[0047] FIG3( b ) is a flow chart of a method for identifying information of items in a refrigerator according to a second embodiment of the present invention.

[0048] FIG4( a ) is a schematic diagram of capturing the space inside a bottle holder in a refrigerator according to a third embodiment of the present invention (except for the outline and the items on both sides for illustration, the rest of the content in the diagram is omitted).

[0049] FIG4( b ) is a flow chart of a method for identifying information of items in a refrigerator according to a third embodiment of the present invention.

[0050] FIG5( a ) is a schematic diagram of capturing the space inside a bottle holder in a refrigerator according to a fourth embodiment of the present invention (except for the outline and the items on both sides for illustration, the rest of the content in the diagram is omitted).

[0051] FIG5( b ) is a flow chart of a method for identifying information of items in a refrigerator according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0054] For ease of explanation, this document uses terms that indicate spatial relative positions, such as "upper," "lower," "rear," "front," etc., to describe the relationship of one unit or feature shown in the drawings relative to another unit or feature. Terms that indicate spatial relative positions may include different orientations of the device during use or operation other than the orientation shown in the drawings. For example, if the device in the drawings is turned over, units described as being "below" or "above" other units or features will be located "below" or "above" the other units or features. Therefore, the exemplary term "below" can encompass both below and above spatial orientations.

[0055] like Figure 1 As shown, the present invention provides a method for identifying information of items in a refrigerator, comprising the steps of:

[0056] S1: Obtain a bird's-eye view image of the space inside the refrigerator bottle holder.

[0057] In this embodiment, the camera is arranged on the top of the bottle holder, and the objects are photographed from above to obtain a bird's-eye view image. In order to avoid mutual occlusion between objects in the bottle holder, which is not conducive to the detection and identification of object information, the present invention preferably arranges the camera in the middle position of the top of the bottle holder.

[0058] Of course, in other embodiments of the present invention, two or more cameras may be provided in the bottle holder to obtain two or more bird's-eye views of the space inside the refrigerator bottle holder taken from different angles.

[0059] S2: Identify a contour frame composed of intersecting contour lines in the bird's-eye view image.

[0060] Since the bottom and wall of the bottle seat are made of different materials, the bottom and wall of the bottle seat captured by the camera usually have obvious pixel grayscale differences in the image, and obvious edge contour lines can be detected and identified.

[0061] Specifically, an embodiment of the present invention uses an edge detection algorithm to identify the edge contour of the bottle seat and calibrate the intersecting contours of the central area of ​​the bird's-eye view image. The intersection of the contours forms a contour frame 1, as shown in FIG. Figure 2(a)-5(a) Here, the edge detection algorithm may adopt currently common algorithms such as Roberts, Sobel, Prewitt, Canny, Log, etc. The specific algorithm contents are existing technologies and will not be described in detail here.

[0062] S3: Segmenting a plurality of detection frames in the bird's-eye view image, each detection frame corresponding to an object.

[0063] When the wall of the bottle holder is not made of frosted material, the objects in the bottle holder will often project corresponding shadows on the wall of the bottle holder, that is, their mirror images. Therefore, the objects in the bird's-eye view image obtained in the present invention include the real objects placed in the bottle holder and the mirror images corresponding to the real objects.

[0064] Specifically, objects in the bird's-eye view image are identified, and the Yolo V3 object detection algorithm is used to calibrate the detection frame 2 corresponding to each object in the bird's-eye view image. The specific content of the Yolo V3 object detection algorithm is prior art and will not be elaborated on here.

[0065] S4: Select a contour line and identify the position of the contour line in the contour frame.

[0066] Specifically, the outline frame 1 is a square structure, which includes an outline line 11 located at the top of the outline frame, an outline line 12 located at the bottom of the outline frame, an outline line 13 located at the left of the outline frame, and an outline line 14 located at the right of the outline frame.

[0067] Illustratively, in the embodiment of the present invention, the selected contour line may be contour line 11 , contour line 12 , contour line 13 or contour line 14 .

[0068] S5: Select a detection line, where the position of the detection line in the detection frame is opposite to the position of the contour line in the contour frame.

[0069] In one embodiment of the present invention, the detection frame 2 is a square structure, which includes a detection line 21 located above the detection frame, a detection line 22 located below the detection frame, a detection line 23 located to the left of the detection frame, and a detection line 24 located to the right of the detection frame.

[0070] Here, the selected detection line may be detection line 21, detection line 22, detection line 23, or detection line 24. In the embodiment of the present invention, the orientation of the selected detection line in the detection frame 2 is opposite to the orientation of the contour line in the contour frame 1. For example, if contour line 11 is selected in step S4, detection line 22 is selected in step S5; if contour line 12 is selected in step S4, detection line 21 is selected in step S5; if contour line 13 is selected in step S4, detection line 24 is selected in step S5; if contour line 14 is selected in step S4, detection line 23 is selected in step S5.

[0071] Furthermore, before "selecting a detection line," the system also includes the step of obtaining detection frames adjacent to the contour line. For example, if contour line 11 is selected in step S4, the system only obtains object detection frames on the two surfaces adjacent to contour line 11. Object detection frames on other surfaces are not considered as a criterion.

[0072] Whether the object identified in the bird's-eye view image is a real object or a mirrored object is determined based on whether the relative orientation of the selected detection line and the contour line is the same as the orientation of the contour line in the contour frame 1 .

[0073] S61: If the relative orientation of the detection line and the contour line is the same as the orientation of the contour line in the contour frame 1, the corresponding object is a mirror image object.

[0074] S62: If the relative orientation of the detection line and the contour line is opposite to the orientation of the contour line in the contour frame 1, the corresponding object is a real object.

[0075] For example, as shown in Figure 2(a), for object a identified in the bird's-eye view image, its detection frame 2a includes 21a located at the top, 22a located at the bottom, 23a located at the left, and 24a located at the right. If contour line 11 is selected in step S4, then detection line 22a is selected in step S5 as the judgment standard in the embodiment of the present invention. Since the relative orientation of detection line 22a and contour line 11 is that detection line 22a is located above contour line 11, and contour line 11 is located above contour frame 1, for object a, the relative orientation of detection line 22a and contour line 11 is the same as the orientation of contour line 11 in contour frame 1. That is, step S61 is executed, and the system determines that object a is a mirrored object.

[0076] For object b identified in the bird's-eye view image, its detection frame 2b includes 21b located at the top, 22b located at the bottom, 23b located at the left, and 24b located at the right. If contour line 11 is selected in step S4, detection line 22b is selected in step S5 as the judgment criterion in the embodiment of the present invention. Since the relative orientation of detection line 22b and contour line 11 is that detection line 22b is located below contour line 11, and contour line 11 is located above contour frame 1, for object b, the relative orientation of detection line 22b and contour line 11 is opposite to the orientation of contour line 11 within contour frame 1. Therefore, step S62 is executed to determine that object b is a genuine object.

[0077] For items c and d in FIG2( a ), according to the above judgment method, it can be judged that item c is a mirrored item and item d is a real item.

[0078] Due to the different selected contour lines and detection lines, the present invention provides four specific methods for identifying items in a refrigerator bottle holder, which are described in detail below using four embodiments.

[0079] Example 1

[0080] In the first embodiment, the contour line 11 is selected, as shown in FIG2(a), and the recognition method (FIG2(b)) includes the following steps:

[0081] S4A1: Select a contour line 11, where the contour line 11 is located on the upper edge of the contour frame 1.

[0082] S5A1: Select a detection line 22 , where the detection line 22 is located at the lower edge of the detection frame 2 .

[0083] S61A1: If the detection line 22 is located above the contour line 11, the corresponding object is a mirror image object.

[0084] S62A1: If the detection line 22 is below the contour line 11, the corresponding object is a real object.

[0085] Similarly, before selecting a detection line 22, the method also includes the step of obtaining detection frames on both sides adjacent to the contour line 11. As shown in Figure 2(a), for item a in the image, since detection line 22a is above contour line 11, item a is determined to be a mirror image. For item b in the image, since detection line 22b is below contour line 11, item b is determined to be a genuine item. For item c in the image, since detection line 22c is above contour line 11, item c is determined to be a mirror image. For item d in the image, since item d is a large item, such as a large bottle of milk or beverage, in the obtained bird's-eye view, the detection frame 2d corresponding to item d spans both sides adjacent to contour line 11. However, since detection line 22d is below contour line 11, item d can also be determined to be a genuine item according to the above method. The following three other methods for determining whether an item is a real object or a mirror image can all be used to identify large items in a bottle holder, without causing differences in recognition results due to differences in object size.

[0086] In this embodiment, the object detection frames on both sides of the contour line 11 obtained in the bird's-eye view can use the object information recognition method in Figure 2(b) to determine whether the object is a real object or a mirrored object.

[0087] Example 2

[0088] In the second embodiment, the contour line 12 is selected, as shown in FIG3(a), and the recognition method (FIG3(b)) includes the steps of:

[0089] S4B1: Select a contour line 12, where the contour line 12 is located at the lower edge of the contour frame 1.

[0090] S5B1: selecting a detection line 21 , where the detection line 21 is located on the upper edge of the detection frame 2 .

[0091] S61B1: If the detection line 21 is located above the contour line 12, the corresponding object is a real object.

[0092] S62B1: If the detection line 21 is located below the contour line 12, the corresponding object is a mirror image object.

[0093] Similarly, before selecting a detection line 21, the process also includes the step of obtaining detection frames on both sides of the contour line 12. As shown in Figure 3(a), for item e in the image, since detection line 21e is above contour line 12, item e is determined to be a genuine item. For item f in the image, since detection line 21f is below contour line 12, item f is determined to be a mirror image.

[0094] In this embodiment, the object detection frames on both sides of the contour line 12 obtained in the bird's-eye view can use the object information recognition method in Figure 3(b) to determine whether the object is a real object or a mirrored object.

[0095] Example 3

[0096] In the third embodiment, the contour line 13 is selected, as shown in FIG4(a), and the identification method (FIG4(b)) includes the steps of:

[0097] S4C1: Select a contour line 13, where the contour line 13 is located on the left line of the contour frame 1.

[0098] S5C1 : Select a detection line 24 , where the detection line 24 is located on the right side of the detection frame 2 .

[0099] S61C1: If the detection line 24 is located to the left of the contour line 13, the corresponding object is a mirror image object.

[0100] S62C1: If the detection line 24 is located to the right of the contour line 13, the corresponding object is a real object.

[0101] Similarly, before selecting a detection line 24, the process also includes obtaining detection frames on both sides of the contour line 13. As shown in Figure 4(a), for item g, since detection line 24g is to the left of contour line 13, item g is determined to be a mirror image. For item h, since detection line 24h is to the right of contour line 13, item h is determined to be a genuine object.

[0102] In this embodiment, the object detection frames on both sides of the contour line 13 obtained in the bird's-eye view can use the object information recognition method in Figure 4(b) to determine whether the object is a real object or a mirrored object.

[0103] Example 4

[0104] In the fourth embodiment, the contour line 14 is selected, as shown in FIG5(a), and the recognition method (FIG5(b)) includes the steps of:

[0105] S4D1: Select a contour line 14, which is located on the right side of the contour frame 1.

[0106] S5D1 ​​: Select a detection line 23 , where the detection line 23 is located on the left line of the detection frame 2 .

[0107] S61D1: If the detection line 23 is located to the left of the contour line 14, the corresponding object is a real object.

[0108] S62D1: If the detection line 23 is located to the right of the contour line 14, the corresponding object is a mirror image object.

[0109] Similarly, before selecting a detection line 23, the process also includes obtaining detection frames on both sides of the contour line 14. As shown in Figure 5(a), for item i in the image, since detection line 23i is located to the left of contour line 14, item i is determined to be a genuine item. For item j in the image, since detection line 23j is located to the right of contour line 14, item j is determined to be a mirror image item.

[0110] In this embodiment, the object detection frames on both sides of the contour line 14 obtained in the bird's-eye view can use the object information recognition method in Figure 5(b) to determine whether the object is a real object or a mirrored object.

[0111] Furthermore, after identifying whether the object in the bird's-eye view is a real object or a mirrored object, the method further includes the steps of:

[0112] S7: Excluding mirrored objects, and counting the number of real objects in the bird's-eye view image.

[0113] The present invention also provides a refrigerator comprising a bottle holder, a camera, a memory and a processor.

[0114] Furthermore, the bottle seat includes a top, a bottom and a wall of the bottle seat, and the camera is arranged in the middle of the top of the bottle seat, vertically downward.

[0115] The bottom of the bottle seat is used for placing articles, and at least one wall surface of the bottle seat is a glass surface.

[0116] The memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the above-mentioned method for identifying information of items in the refrigerator are implemented.

[0117] The present invention also provides a computer storage medium, which stores a computer program, and when the computer program is run, the device where the computer storage medium is located executes the steps of the above-mentioned method for identifying information of items in a refrigerator.

[0118] In summary, the present invention uses the edge detection algorithm and the Yolo V3 target detection algorithm to calibrate the contour box and the detection box corresponding to the object, and obtains the contour line and the detection line. The orientation of the detection line in the detection box is opposite to the orientation of the contour line in the contour box. By comparing whether the relative orientation of the detection line and the contour line is the same as the orientation of the contour line in the contour box, it is determined whether the object detected in the bird's-eye view image is a real object or a mirrored object. In this way, the interference of the mirrored object can be eliminated and the accuracy of the recognition algorithm can be improved.

[0119] It should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation method can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0120] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying information of items in a refrigerator, characterized in that: Including steps: Controlling a camera located in the middle of the top of the refrigerator bottle holder to capture a bird's-eye view image of the space inside the refrigerator bottle holder; Using an edge detection algorithm, calibrate the intersecting contour lines of the central area of ​​the bird's-eye view image, wherein the intersecting contour lines form a contour frame; Segmenting a plurality of detection frames in the bird's-eye view image, each detection frame corresponding to an object; Selecting a contour line and identifying the position of the contour line within the contour frame; Selecting a detection line, wherein the position of the detection line in the detection frame is opposite to the position of the contour line in the contour frame; If the relative orientation of the detection line to the contour line is the same as the orientation of the contour line within the contour frame, the corresponding object is a mirror image object; If the relative orientation of the detection line and the contour line is opposite to the orientation of the contour line in the contour frame, the corresponding object is a real object.

2. The method for identifying information of items in a refrigerator according to claim 1, wherein: “Segmenting a plurality of detection frames in the bird’s-eye view image, each detection frame corresponding to an object” specifically includes: Using the Yolo V3 object detection algorithm, calibrate the detection box corresponding to each object in the bird's-eye view image; The objects in the bird's-eye view image include real objects placed in the bottle holder and mirror-image objects corresponding to the real objects.

3. The method for identifying information of items in a refrigerator according to claim 2, characterized in that: Before "selecting a detection line, wherein the position of the detection line in the detection frame is opposite to the position of the contour line in the contour frame", the method further includes the following steps: Get the detection boxes on both sides adjacent to the contour line.

4. The method for identifying information of items in a refrigerator according to claim 3, wherein: The method specifically includes: Selecting a contour line, where the contour line is located at the upper edge of the contour frame; Selecting a detection line, where the detection line is located at the lower edge of the detection frame; If the detection line is above the contour line, the corresponding object is a mirror image object; If the detection line is below the contour line, the corresponding object is a real object.

5. The method for identifying information of items in a refrigerator according to claim 3, wherein: The method specifically includes: Selecting a contour line, the contour line is located at the lower edge of the contour frame; Selecting a detection line, where the detection line is located at the upper edge of the detection frame; If the detection line is above the contour line, the corresponding object is a real object; If the detection line is below the contour line, the corresponding object is a mirror image object.

6. The method for identifying information of items in a refrigerator according to claim 1, wherein: The method further comprises the steps of: Excluding mirrored objects, the number of real objects in the bird's-eye view image is counted.

7. A refrigerator comprising: Bottle holder, camera, memory and processor, characterized in that, The bottle seat comprises a top, a bottom and a wall of the bottle seat, and the camera is arranged in the middle of the top of the bottle seat and is arranged vertically downward; The bottom of the bottle seat is used for placing items; At least one wall surface of the bottle base is a glass surface; The memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the method for identifying information of items in a refrigerator as described in any one of claims 1 to 6 are implemented.

8. A computer storage medium storing a computer program, characterized in that: When the computer program is executed, the device where the computer storage medium is located executes the steps of the method for identifying information of items in a refrigerator according to any one of claims 1 to 6.

Images