Refrigerator internal item information recognition method, refrigerator, and computer storage medium

Abstract

The application discloses a refrigerator, a refrigerator article information recognition method and a computer storage medium. The method comprises the following steps: obtaining an aerial image of a space in a refrigerator bottle seat; positioning a character region on an upper surface of each article in the aerial image; constructing a recognition frame corresponding to the character region on the upper surface of an article, each recognition frame containing all character information in the character region corresponding to the recognition frame; extracting the recognition frame, keeping the shape of the recognition frame unchanged, and splicing each recognition frame; constructing a to-be-recognized region containing all recognition frames; and recognizing character information in the to-be-recognized region. The application can output multiple character information through one recognition algorithm, thereby improving the efficiency of the recognition algorithm.

Description

Technical Field

[0001] This invention relates to the field of refrigeration devices, and more particularly to a method for identifying information about items inside a refrigerator, a refrigerator, and a computer storage medium. Background Technology

[0002] With the development of smart appliances, the ability to identify items inside has become an essential function of smart refrigerators. Typically, to identify items stored inside the refrigerator, one or more cameras are installed to capture images of the items. These images are then processed and sent to the user's device for viewing. Placing the cameras above the items avoids the problem of items obstructing each other, thus improving the accuracy of item identification.

[0003] However, many problems still exist in the actual implementation of smart refrigerators. For example, how to identify the text information of items inside the refrigerator's bottle holder, including item names, production dates, and expiration dates. Current methods involve inputting the entire image of the refrigerator's bottle holder into the algorithm, locating the specific position of the item's text information on the image, identifying that item's text information, and then proceeding to identify the text information of the next item. When there are many items in the refrigerator's bottle holder, the time required for this method of identifying each item increases exponentially, resulting in low recognition efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a method for identifying information about items inside a refrigerator, a refrigerator, and a computer storage medium.

[0005] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides a method for identifying information about items inside a refrigerator, comprising the following steps:

[0006] Obtain an aerial view of the space inside the refrigerator's bottle holder;

[0007] Locate the character regions on the upper surface of each item in the bird's-eye view image;

[0008] Construct a recognition box, wherein the recognition box corresponds to the character area on the surface of an object, and each recognition box contains all the character information within the corresponding character area;

[0009] Extract the recognition boxes, keep the shape of the recognition boxes unchanged, and stitch the recognition boxes together;

[0010] Construct the region to be recognized, containing all the bounding boxes;

[0011] Identify character information within the area to be identified.

[0012] As a further improvement to one embodiment of the present invention, locating the character region on the upper surface of the object in the bird's-eye view image specifically includes:

[0013] Using a character localization algorithm, the character region on the upper surface of the object in the bird's-eye view image is located.

[0014] As a further improvement to one embodiment of the present invention, the step of extracting the recognition frame, keeping the shape of the recognition frame unchanged, and splicing the recognition frames specifically includes:

[0015] While maintaining the relative positional relationship of each of the recognition boxes in the bird's-eye view image, extract each of the recognition boxes;

[0016] One identification frame is kept in a fixed position, and all other identification frames are moved toward the same identification frame in sequence for splicing. The movement stops when the identification frame comes into contact with another identification frame.

[0017] As a further improvement to one embodiment of the present invention, the step of extracting the recognition frame, keeping the shape of the recognition frame unchanged, and splicing the recognition frames specifically includes:

[0018] All recognition boxes are then tightly spliced ​​together horizontally and vertically.

[0019] As a further improvement to one embodiment of the present invention, the construction of the region to be identified, including the recognition box, specifically includes:

[0020] Construct the region to be identified with a rectangular structure;

[0021] The maximum value of the overall length of all the spliced ​​recognition frames in the horizontal direction is taken as the length of the region to be recognized, and the maximum value of the overall length of all the spliced ​​recognition frames in the vertical direction is taken as the width of the region to be recognized.

[0022] As a further improvement of one embodiment of the present invention, the step of recognizing the character information of the region to be recognized specifically includes:

[0023] Using an OCR recognition algorithm, all character information within the area to be recognized is identified.

[0024] As a further improvement to one embodiment of the present invention, the method further includes:

[0025] The recognition results are mapped one by one to each recognition box to obtain the character information in each recognition box;

[0026] By mapping the character regions on the upper surface of the objects in the bird's-eye view image, the character information of the upper surface of each object in the bird's-eye view image is obtained.

[0027] As a further improvement to one embodiment of the present invention, the step of mapping the recognition results one-to-one to each recognition box to obtain the character information within each recognition box; and mapping them to the character areas on the upper surfaces of the objects in the bird's-eye view image to obtain the character information on the upper surfaces of each object in the bird's-eye view image, specifically includes:

[0028] Detect the date information in the character information, map the date information to each recognition box, and obtain the date information in each recognition box;

[0029] Based on the character area on the upper surface of each item in the bird's-eye view image corresponding to each recognition box, the date information on the upper surface of each item in the bird's-eye view image is obtained.

[0030] This invention provides a refrigerator, comprising: a bottle holder, a camera, a memory, and a processor, wherein,

[0031] The camera is located at the top center of the bottle holder, facing vertically downwards;

[0032] The bottom of the bottle holder is used to place items;

[0033] The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps of the refrigerator item information identification method as described in any of the above embodiments.

[0034] The present invention also provides a computer storage medium storing a computer program, wherein the computer program, when executed, causes the device containing the computer storage medium to perform the steps of the refrigerator item information identification method according to any of the above embodiments.

[0035] Compared with the prior art, the present invention constructs a recognition box for the character area on the surface of each item in the bottle holder, extracts all the recognition boxes constructed in the bird's-eye view image and keeps their shape unchanged, splices the recognition boxes to form a recognition area, and then recognizes the character information in the recognition area. Multiple character information can be output through a single recognition algorithm, thus improving the efficiency of the recognition algorithm. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a refrigerator item information identification method according to one embodiment of the present invention.

[0037] Figure 2 This is a simplified schematic diagram of an aerial view of the inside of a refrigerator bottle holder obtained by taking a picture according to one embodiment of the present invention.

[0038] Figure 3-7 This is a schematic diagram of the identification box splicing process extracted in one embodiment of the present invention.

[0039] Figure 8This is a schematic diagram of the structure for constructing the region to be identified in one embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0042] For ease of explanation, this document uses terms indicating relative spatial position, such as "above," "below," "behind," and "front," to describe the relationship of one unit or feature shown in the accompanying drawings relative to another unit or feature. Terms indicating relative spatial position can include different orientations of the device during use or operation besides those shown in the figures. For example, if the device in the figures is flipped, a unit described as being "below" or "above" other units or features will be located "below" or "above" other units or features. Therefore, the exemplary term "below" can encompass both "below" and "above" spatial orientations.

[0043] Because the production date and shelf life information of food items in the refrigerator are scattered and not fixed in location, some are even placed at the bottom of the food (for example, the production date of some yogurt is placed at the bottom), it is inconvenient to identify and manage these food items uniformly. Therefore, when food items are stored in the refrigerator, a label with a uniform format containing basic information about the food can be printed for them, and the label can be placed in a fixed position on the food (for example, pasted on the top of the food). In this way, the computer and the user can obtain information such as the name and production date of the food through the label, making it convenient for the computer and the user to manage the food.

[0044] like Figure 1 As shown, this embodiment provides a method for identifying information about items inside a refrigerator, including the following steps:

[0045] S1: Obtain a bird's-eye view of the space inside the refrigerator bottle holder 1.

[0046] In this embodiment, the camera is located at the top of the bottle holder to capture a bird's-eye view of the items. To avoid the items inside the bottle holder from obstructing each other and hindering the detection and identification of item information, the present invention preferably places the camera at the middle of the top of the bottle holder.

[0047] Of course, in other embodiments of the present invention, two or more cameras may be provided inside the bottle holder to obtain two or more bird's-eye view images of the space inside the refrigerator bottle holder taken from different angles, or the camera may be set at the bottom of the bottle holder, vertically downwards, to capture information about the items inside the bottle holder below it.

[0048] S2: Locate the character area on the upper surface of each item in the bird's-eye view image.

[0049] For example, such as Figure 2 The image shown is a bird's-eye view of the space inside the refrigerator bottle holder obtained in this embodiment. In practical applications, the refrigerator bottle holder can generally only hold two rows of items in the horizontal direction. As mentioned above, in order to facilitate the identification and unified management of the food items inside the refrigerator bottle holder, information such as the type of item or the production date and shelf life of the item is usually printed and placed on the top cover of the item.

[0050] Specifically, this embodiment utilizes a character localization algorithm to locate the character region on the surface of an item in a bird's-eye view image. In other words, the algorithm can directly pinpoint the exact location of textual information about the item's type and numerical information such as production date and shelf life on the surface of an item placed within the bottle holder. The character localization algorithm used here is existing technology and will not be elaborated upon further.

[0051] Furthermore, before step S2, the method includes the step of locating the position of the items in the bird's-eye view image and identifying the item information. In the subsequent identification method, after obtaining the specific character information on the surface of each item, it can be associated with the item type information corresponding to each character area and the position information of the item within the bottle holder, thereby obtaining the item information placed at different positions within the bottle holder, including item type information, item numerical information, etc.

[0052] Specifically, the YOLO V4 object detection algorithm can be used to mark the bounding box corresponding to each object in the bird's-eye view image. The position of the object within each bounding box in the bottle holder can be obtained based on its marked position. Within each bounding box, the outer contour type of the object is detected and identified. Object information is detected and identified using algorithms such as YOLO, SSD, or traditional detection algorithms. Based on the top image of the object, a deep learning algorithm is used to identify the object category information, thus obtaining the type information of the object corresponding to each bounding box.

[0053] For example, in this embodiment, articles can be divided into bottled articles, canned articles, boxed articles, and other articles. Bottled articles have bottle caps and are articles with obvious bottle caps, such as bottled beverages. Canned articles have a circular outer outline in their top view and are articles with a basically cylindrical outer packaging, such as canned beverages and sauces. Boxed articles have a quadrilateral shape in their top view and are articles with a basically cuboid or cube outer packaging, such as boxed dairy products. Articles that do not belong to the above three categories are classified as other articles.

[0054] like Figure 2 In the bird's-eye view shown, items a1, a2, and a3 are bottled items, items b1, b2, and b3 are canned items, and items c1 and c2 are boxed items.

[0055] S3: Construct recognition box 1, which corresponds to the character area on the upper surface of an item, and each recognition box 1 contains all the character information within the corresponding character area.

[0056] Specifically, after locating the character areas on the surface of each item, a rectangular recognition box 1 is constructed for each character area. Each recognition box 1 contains all the information within its corresponding character area.

[0057] S4: Extract the recognition box 1, keep the shape of the recognition box 1 unchanged, and splice the recognition boxes 1 together.

[0058] Because it is often difficult to position the labels with text on the surface of various items in a fixed orientation when placing them in the bottle holder, the recognition frames 1 corresponding to the character areas on the surface of each item have different angles. For item c1 in the figure, the recognition frame c11 is a rectangular structure constructed horizontally, while the recognition frame c21 of item c2, placed to the right of item c1, is a rectangular structure constructed vertically. The recognition frames 1 constructed for the character areas on the surface of bottled and canned items in the figure are constructed at a certain angle relative to the recognition frames c11 and c21 of items c1 and c2, respectively.

[0059] like Figure 3 As shown, the recognition boxes 1 constructed from the character regions on the upper surface of each object in the bird's-eye view image are extracted. The shape, position, and relative tilt angle of each recognition box 1 remain unchanged. Each recognition box 1 is placed on the same recognition plane and the algorithmic recognition of unified character information is performed.

[0060] To further improve the efficiency of the recognition algorithm, the size of the area to be detected and recognized can be further reduced. In a specific embodiment of the present invention, all recognition boxes 1 are arranged in the order of their arrangement in the bird's-eye view image. The position of one recognition box is fixed and all other recognition boxes are moved toward the direction of the recognition box in sequence for splicing. The movement stops when the recognition box is in contact with another recognition box.

[0061] For example, such as Figure 4-7 As shown, the position of the recognition frame c11 corresponding to item c1 can be kept unchanged first. Then, the recognition frame c21 of item c2 can be moved towards the recognition frame c11 until the recognition frame c21 contacts the recognition frame c11. Next, the recognition frame b11 of item b1 can be moved towards the recognition frame c21 until it contacts the recognition frame c21. Finally, the recognition frame a11 of item a1 can be moved towards the recognition frame b11 until it contacts the recognition frame b11. This forms the splicing area A.

[0062] Using the same method, control the movement and splicing of the recognition frames 1 corresponding to the items in the lower row of the bottle holder. First, keep the position of the recognition frame b21 corresponding to item b2 stationary. Then, control the recognition frame a21 corresponding to item a2 to move towards recognition frame b21 until it contacts b21. Next, control the recognition frame b31 corresponding to item b3 to move towards recognition frame a21 until it contacts a21. Finally, control the recognition frame a31 corresponding to item a3 to move towards recognition frame b31 until it contacts b31. This forms the splicing area B.

[0063] Keeping stitching area A stationary, move stitching area B toward stitching area A until a recognition box in stitching area B contacts a recognition box in stitching area A, then stop moving stitching area B. Figure 7 As shown, the movement of the splicing area B stops when the recognition box b21 comes into contact with the recognition box c11.

[0064] Of course, it is also possible to keep any one of the recognition frames in the same row stationary, control the other recognition frames to move toward the recognition frame to form splicing areas A and B, and then splice the recognition frames in splicing areas A and B together. This invention does not limit this.

[0065] S5: Construct the region to be recognized 2 containing all recognition boxes 1.

[0066] Specifically, a rectangular region 2 to be identified is constructed. The maximum value of the overall length of all the spliced ​​recognition boxes in the horizontal direction is taken as the length l of the region 2 to be identified, and the maximum value of the overall length of all the spliced ​​recognition boxes in the vertical direction is taken as the width w of the region 2 to be identified.

[0067] like Figure 8As shown, the horizontal distance between the leftmost recognition box c11 and the rightmost recognition box a31 in the splicing area is taken as the length l of the area to be recognized 2, and the vertical distance between the bottommost recognition box a21 and the topmost recognition box a11 in the splicing area is taken as the width w of the area to be recognized 2.

[0068] Of course, the smaller the area of ​​the constructed region to be identified 2, the higher the algorithmic recognition efficiency of the region to be identified 2 in subsequent operations. Therefore, in another embodiment of the present invention, all recognition boxes 1 are sequentially and tightly spliced ​​together horizontally and vertically. Specifically, in this embodiment, the arrangement order of each recognition box 1 does not need to be arranged according to its relative position relationship in the bird's-eye view image. It is only necessary to ensure that there is at least point contact between each recognition box 1 and the recognition box 1 it is spliced ​​with, and that there is no overlapping area. Then, the region to be identified 2 is constructed from the spliced ​​area.

[0069] Furthermore, in this embodiment, when all the recognition boxes 1 are arranged horizontally and then tightly spliced ​​together, and the other recognition boxes 1 are arranged vertically and then tightly spliced ​​together, so that the area of ​​the constructed region to be recognized 2 is minimized, this is the optimal recognition scheme of this application.

[0070] S6: Identify the character information within the region to be identified 2.

[0071] Specifically, the constructed recognition area 2 is input into the system for algorithm recognition. The OCR recognition algorithm is used to recognize all the character information in the recognition area 2.

[0072] Furthermore, the identification method in this embodiment also includes the following steps:

[0073] S7: Map the recognition results one by one to each recognition box 1 to obtain the character information in each recognition box 1.

[0074] S8: Corresponding to the character area on the upper surface of the object in the bird's-eye view image, obtain the character information on the upper surface of each object in the bird's-eye view image.

[0075] The character information in the recognition results specifically includes text information and numerical information on the surface of the item, with the numerical information being the date information on the surface of the item.

[0076] Specifically, after obtaining all character information within the region to be recognized 2, the content related to date information in all character information is detected, and the date information is mapped to each recognition box 1 to obtain the date information within each recognition box 1.

[0077] More specifically, based on the character area on the upper surface of each item in the bird's-eye view image corresponding to each recognition box 1, the date information on the upper surface of each item in the bird's-eye view image is obtained.

[0078] By combining the information on the location of each item in the bottle holder and the type of each item, the system can push data such as the type, location, and date (production date, shelf life, etc.) of each item placed in the refrigerator bottle holder to the user.

[0079] The present invention also provides a refrigerator, which includes a bottle holder, a camera, a memory, and a processor.

[0080] Furthermore, the bottle holder includes a top, a bottom, and a wall surface, and the camera is located at the center of the top of the bottle holder, vertically downwards.

[0081] The bottom of the bottle holder is used to place items, and the memory stores a computer program that can run on the processor. When the processor executes the program, it implements the steps of the refrigerator item information identification method in any of the above embodiments.

[0082] The present invention also provides a computer storage medium storing a computer program, wherein the computer program, when executed, causes the device on which the computer storage medium is located to perform the steps of any of the above-described refrigerator item information identification methods.

[0083] In summary, this invention constructs a recognition box for the character areas on the surface of each item in the bottle holder, extracts all the recognition boxes constructed in the bird's-eye view image while keeping their shapes unchanged, and splices the recognition boxes to form a recognition area. Then, the character information in the recognition area is recognized. Multiple character information can be output through a single recognition algorithm, thus improving the efficiency of the recognition algorithm.

[0084] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0085] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying information about items inside a refrigerator, characterized in that, Including the following steps: Obtain an aerial view of the space inside the refrigerator's bottle holder; Locate the character regions on the upper surface of each item in the bird's-eye view image; Construct a recognition box, wherein the recognition box corresponds to the character area on the surface of an item, and each recognition box contains all the character information within the corresponding character area; Extract the recognition box, keep the shape of the recognition box unchanged, keep the relative position relationship of each recognition box in the bird's-eye view image unchanged, fix the position of one recognition box, and move all other recognition boxes toward the recognition box in sequence for stitching. Stop moving when the recognition box touches another recognition box, and stitch the recognition boxes together. Construct the region to be recognized, containing all the bounding boxes; Identify character information within the area to be identified; The step of extracting the recognition boxes while maintaining their shape and then stitching them together specifically includes: All recognition boxes are tightly spliced ​​together horizontally and vertically in sequence; The method further includes: The recognition results are mapped one by one to each recognition box to obtain the character information in each recognition box; By mapping the character regions on the upper surfaces of objects in the bird's-eye view image, character information of the upper surfaces of each object in the bird's-eye view image is obtained; The process of mapping the recognition results one-to-one to each recognition box to obtain the character information within each recognition box, and mapping them to the character areas on the upper surface of the objects in the bird's-eye view image to obtain the character information on the upper surface of each object in the bird's-eye view image, specifically includes: Detect the date information in the character information, map the date information to each recognition box, and obtain the date information in each recognition box; Based on the character area on the upper surface of each item in the bird's-eye view image corresponding to each recognition box, the date information on the upper surface of each item in the bird's-eye view image is obtained.

2. The method for identifying information about items inside a refrigerator according to claim 1, characterized in that, Locating the character region on the upper surface of the object in the bird's-eye view specifically includes: Using a character localization algorithm, the character region on the upper surface of the object in the bird's-eye view image is located.

3. The method for identifying information about items inside a refrigerator according to any one of claims 1 to 2, characterized in that, The construction of the region to be identified, including the recognition box, specifically includes: Construct the region to be identified with a rectangular structure; The maximum length of the entire horizontal dimension of all the spliced ​​recognition frames is taken as the length of the region to be recognized, and the maximum length of the entire vertical dimension of all the spliced ​​recognition frames is taken as the width of the region to be recognized.

4. The method for identifying information about items inside a refrigerator according to claim 3, characterized in that, The identification of character information in the region to be identified specifically includes: Using an OCR recognition algorithm, all character information within the area to be recognized is identified.

5. The method for identifying information about items inside a refrigerator according to claim 1, characterized in that, Item information includes item type information; The method further includes: The detection box corresponding to each item in the bird's-eye view image is identified using an object detection algorithm; The position of the item within the detection frame in the refrigerator bottle holder is obtained based on the calibrated position of each detection frame; Within each detection frame, the outer contour type of the item is detected and identified, and the item category information is identified.

6. A refrigerator, comprising: Bottle holder, camera, memory and processor, characterized in that, The camera is located at the top center of the bottle holder, facing vertically downwards; The bottom of the bottle holder is used to place items; The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps of the refrigerator item information identification method as described in any one of claims 1-5.

7. A computer storage medium storing a computer program, characterized in that, When the computer program is executed, it causes the device containing the computer storage medium to perform the steps of the refrigerator item information identification method according to any one of claims 1-5.

Images