Image recognition method, device and system

Abstract

The application discloses an image recognition method, device and system. The method comprises the following steps: collecting a food image of food in a refrigerator through a camera installed in the refrigerator, wherein the food image comprises a food label; sending the food image to a cloud server; receiving a first recognition result returned by the cloud server, wherein the first recognition result comprises a result obtained by recognizing the food label in the food image; and displaying a food recognition result based on the first recognition result. The application solves the technical problems of complicated operation and poor user experience in food information input in the related art.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to an image recognition method, apparatus, and system. Background Technology

[0002] Currently, timely determination of whether food stored in the refrigerator has expired is a problem that needs to be solved. In related technologies, the production date and shelf life of food are generally recorded by the user for subsequent reminders. However, when using the above solution to solve the problem, the data entry operation is cumbersome, resulting in a poor user experience.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides an image recognition method, apparatus, and system to at least solve the technical problems of cumbersome operation and poor user experience when entering food information in related technologies.

[0005] According to one aspect of the present invention, an image recognition method is provided, comprising: acquiring a food image of food inside a refrigerator using a camera installed inside the refrigerator, wherein the food image includes a food label; sending the food image to a cloud server; receiving a first recognition result returned by the cloud server, wherein the first recognition result includes a result obtained after recognizing the food label in the food image; and displaying the food recognition result based on the first recognition result.

[0006] Optionally, displaying a food recognition result based on the first recognition result includes: recognizing the food image through the refrigerator's processor to obtain a second recognition result; determining the food recognition result of the food image based on the first recognition result and the second recognition result, and displaying the food recognition result.

[0007] Optionally, the identification result includes at least one of the following: the type of food, the name of the food, the identifier of the food, the quantity of the food, the shelf life of the food, and the storage environment information of the food.

[0008] Optionally, if the food identification result includes the quantity of the food and the shelf life of the food, the method further includes: collecting the available capacity of the area in the refrigerator where the food is stored using a capacity sensor installed in the refrigerator; determining the number of expired food items based on the shelf life; determining the replenishable quantity of food allowed to be stored in the area based on the available capacity, the number of expired food items, and the number of identified food items, and displaying the replenishable quantity.

[0009] Optionally, the method further includes sending a prompt message to a terminal bound to the refrigerator, wherein the prompt message includes the food identification result.

[0010] Optionally, the food image is an image captured by mounting the camera below the refrigerator shelf.

[0011] According to another aspect of the present invention, an image recognition method is also provided, comprising: receiving a food image sent by a refrigerator, wherein the food image is acquired by a camera installed in the refrigerator, and the food image includes a food label; recognizing the food image to obtain a first recognition result, wherein the first recognition result includes the result obtained after recognizing the food label in the food image; and sending the first recognition result to the refrigerator, for the refrigerator to display the food recognition result based on the first recognition result.

[0012] Optionally, the food image is identified to obtain a first identification result, including: identifying the food image using a quantity recognition model to obtain the quantity of the food, wherein the quantity recognition model is trained using multiple sets of data, the multiple sets of data including: an image, the quantity of food in the image; and identifying the food label in the food image using an optical character recognition (OCR) method to obtain the shelf life of the food; wherein the first identification result includes the type of the food, the name of the food, the identifier of the food, the quantity of the food, the shelf life of the food, and the storage environment information of the food.

[0013] Optionally, the step of recognizing the food label in the food image using the OCR method to obtain the shelf life of the food includes: recognizing the text of the food label in the food image using the OCR method to obtain a text recognition result; if the text recognition result includes a shelf life, matching the production date and shelf life of the food based on the text recognition result; and / or, if the text recognition result does not include the shelf life, obtaining the category information of the food, and calculating the shelf life of the food based on the category information and the production date matched from the text recognition result.

[0014] According to another aspect of the present invention, an image recognition method is also provided, comprising: a refrigerator acquiring a food image of food inside the refrigerator using a camera installed inside the refrigerator, wherein the food image includes a food label; the refrigerator sending the food image to a cloud server; the cloud server recognizing the food image to obtain a first recognition result and returning the first recognition result to the refrigerator, wherein the first recognition result includes the result obtained after recognizing the food label in the food image; the refrigerator recognizing the food image to obtain a second recognition result; and the refrigerator determining a food recognition result of the food image based on the first recognition result and the second recognition result, and displaying the food recognition result.

[0015] According to another aspect of the present invention, an image recognition device is also provided, comprising: a acquisition module for acquiring food images of food inside a refrigerator using a camera installed inside the refrigerator, wherein the food images include food labels; a first sending module for sending the food images to a cloud server; a first receiving module for receiving a first recognition result returned by the cloud server, wherein the first recognition result includes the result obtained after recognizing the food labels in the food images; and a display module for displaying the food recognition result based on the first recognition result.

[0016] According to another aspect of the present invention, an image recognition device is also provided, comprising: a receiving module for receiving a food image sent by a refrigerator, wherein the food image is acquired by a camera installed inside the refrigerator; a recognition module for recognizing the food image to obtain a first recognition result, wherein the first recognition result includes the result obtained after recognizing a food label in the food image; and a second sending module for sending the first recognition result to the refrigerator, wherein the refrigerator displays the food recognition result based on the first recognition result.

[0017] According to another aspect of the present invention, an image recognition system is also provided, comprising: a refrigerator and a cloud server, wherein the refrigerator is configured to acquire food images of food inside the refrigerator via a camera installed inside the refrigerator, and send the food images to the cloud server, wherein the food images include food labels; the cloud server is configured to recognize the food images, obtain a first recognition result, and return the first recognition result to the refrigerator, wherein the first recognition result includes the result obtained after recognizing the food labels in the food images; the refrigerator is further configured to recognize the food images, obtain a second recognition result; and determine a food recognition result of the food images based on the first recognition result and the second recognition result, and display the food recognition result.

[0018] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the image recognition methods described herein.

[0019] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the image recognition methods described herein.

[0020] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the image recognition methods described herein.

[0021] In this embodiment of the invention, by acquiring food images and receiving the first recognition result of the food images from the cloud server, the purpose of recognizing food information in the food images is achieved, thereby achieving the technical effect of recognizing food information inside the refrigerator. This solves the technical problems of cumbersome operation and poor user experience when entering food information in related technologies. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0023] Figure 1 This is a flowchart of an image recognition method according to an embodiment of the present invention;

[0024] Figure 2 This is a flowchart of image recognition method two according to an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of image recognition method three according to an embodiment of the present invention;

[0026] Figure 4 This is a design drawing of a refrigerator in the image recognition method of an optional embodiment of the present invention;

[0027] Figure 5a This is a flowchart of an image recognition method according to an optional embodiment of the present invention;

[0028] Figure 5b This is a flowchart of an optional embodiment of the extended image recognition method of the present invention;

[0029] Figure 6 This is a schematic diagram of an image captured according to an optional embodiment of the present invention;

[0030] Figure 7 This is a schematic diagram illustrating OCR recognition of an image after it has been captured, according to an optional embodiment of the present invention.

[0031] Figure 8 This is a structural block diagram of an image recognition device according to an embodiment of the present invention;

[0032] Figure 9 This is a structural block diagram of an image recognition device 2 according to an embodiment of the present invention;

[0033] Figure 10 This is a structural block diagram of an image recognition system according to an embodiment of the present invention;

[0034] Figure 11 This is a structural block diagram of a terminal according to an exemplary embodiment;

[0035] Figure 12 This is a structural block diagram of a server according to an exemplary embodiment. Detailed Implementation

[0036] According to an embodiment of the present invention, an embodiment of an image recognition method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] Figure 1 This is a flowchart of an image recognition method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0038] Step S102: Using a camera installed inside the refrigerator, food images of the food inside the refrigerator are captured, wherein the food images include food labels.

[0039] Step S104: Send the food image to the cloud server;

[0040] Step S106: Receive the first recognition result returned by the cloud server, wherein the first recognition result includes the result obtained after recognizing the food label in the food image;

[0041] Step S108: Based on the first identification result, display the food identification result.

[0042] Through the above steps, the refrigerator side achieves the goal of identifying food information in food images by collecting food images and receiving the recognition results from the cloud server. This achieves the technical effect of identifying food information inside the refrigerator, thereby solving the technical problems of cumbersome operation and poor user experience when entering food information in related technologies.

[0043] As an optional embodiment, a camera installed inside the refrigerator captures images of the food inside, including food labels. The food images are captured by the camera mounted below the refrigerator shelves. Depending on the refrigerator model, the camera can be installed in different locations, and the number of cameras can be adjusted according to the actual situation. The camera can be installed in any location that can obtain valid information about the food, such as under the shelves or drawers inside the refrigerator door to capture images of the food inside the lower shelves or drawers. Other locations where food can be placed can also be used to capture images. Because refrigerator models vary, if the shelves or drawers are too wide or too long, a single camera may not be able to capture images of all the food. A certain number of cameras can be installed under the same shelf or drawer to ensure that images of all the food inside the lower shelves or drawers are captured without repetition, preventing omissions and ensuring that the recorded number of food items is not incomplete, while also preventing duplication and ensuring that the recorded number of food items is greater than the actual number, etc. Furthermore, the camera can respond to capture images in various ways, such as capturing images based on the refrigerator's on / off status, capturing images at predetermined intervals, etc. In scenarios where the camera captures images based on the refrigerator's on / off status, it can respond to the refrigerator's transition from an open to a closed state. Depending on the refrigerator type, different shooting methods can be selected. For example, common refrigerator types include those with constantly lit lights and those where the light turns on when the door is opened and off when it's closed. In scenarios with constantly lit refrigerators, the camera can directly capture images due to ample light. For refrigerators where the light turns on when the door is opened and off when it's closed, the camera needs to activate its flash to capture the image. Different modes can be set to acquire images of the food inside the refrigerator based on different needs and individual refrigerator differences, making it applicable to various scenarios and meeting different requirements.

[0044] As an optional embodiment, after capturing an image but before uploading it to the cloud server, the acquired image can be preprocessed. This preprocessing may include at least one of the following: adjusting the image tone, such as brightness and contrast; adjusting the image size, such as enlarging or reducing the image, adjusting its length, width, and height, cropping a portion of the image, etc. For example, when capturing an image of food inside a refrigerator, if only one food item is present on the shelf or in the drawer, and the image shows only the empty area of ​​the shelf or drawer, the image can be resized, and the portion containing the food item can be cropped. If the food item is too tall and close to the camera, the captured image will be too large, and the image can be reduced in size. If the food item is too short and far from the camera, the captured image will be too small, and the image can be enlarged, etc. Preprocessing the image before uploading it to the cloud server allows for better recognition by the cloud server.

[0045] As an optional embodiment, when capturing food images inside the refrigerator, various types of food can be captured, such as food, beverages, fruits, etc. Different methods can be used to identify the food based on its characteristics. The captured food has certain characteristics that allow it to be subsequently identified by the cloud server. For example, the food may include a food label; for instance, when the food is a bottled beverage, the top of the cap has an expiration date to identify its shelf life. When the food has other characteristics, for example, when the food is boxed milk, the top of the milk has a QR code to scan and retrieve the product information. For example, when the food is a can of a specific shape, the top of the can has an expiration date to identify its shelf life. Furthermore, by recognizing the specific shape of the can, other product information can be determined, such as the brand, type, etc. Specifically, when the food is fruit and the fruit is unpackaged, its shelf life can be estimated based on the typical shelf-life date of that type of fruit, and so on.

[0046] As an optional embodiment, food images are sent to a cloud server. The food images captured by the camera are sent to the cloud server for recognition. The user directly receives the first recognition result returned by the cloud server, which includes the result obtained after recognizing the food label in the food image. Based on the first recognition result, the food recognition result is displayed. For the refrigerator side, only taking a food image and uploading it to the cloud server is required. After a short wait, the recognition result is obtained, and by displaying the food recognition result, the user can know the status of the food in the refrigerator. This significantly reduces the time required for manual data entry, avoids failures, repetitions, and omissions in manual data entry, and obtains more effective information. It provides a simple and convenient way to input food information; users do not need to perform any other operations. They only need to place the food in the refrigerator to obtain the food information, greatly improving the user experience.

[0047] As an optional embodiment, displaying the food identification result based on the first identification result includes: identifying a food image through the refrigerator's processor to obtain a second identification result; determining the food identification result of the food image based on the first and second identification results, and displaying the food identification result. The first identification result is based on the identification result from the cloud server, and the second identification result is based on the identification result from the refrigerator's processor. The food identification result is determined by combining the first and second identification results, comprehensively considering both remote and local identification results to determine the final food identification result. On the one hand, dual identification improves the accuracy of identification. On the other hand, the two identification methods can complement or confirm each other, making the identification comprehensive.

[0048] It should be noted that the recognition results can include different information depending on the food image captured. Various recognition methods can be used to obtain the results, such as artificial intelligence image recognition, OCR (Optical Character Recognition), and so on. Typically, a single captured food image may contain multiple food items, so the recognition result can include the quantity of food. The image can also provide the shelf life of the corresponding food item. If the captured food image has certain characteristics, such as food containing QR codes or food with specific shapes, more results can be obtained, such as the type of food, the brand, the ingredients, the method of consumption, the name of the food, the identification mark, and the storage environment information. Based on the characteristics of the food, more food recognition results can be obtained about the food in the image.

[0049] As an optional embodiment, when the food identification result includes the quantity and shelf life of the food, the method further includes: collecting the available capacity of the food storage area in the refrigerator using a capacity meter installed inside the refrigerator; determining the number of expired foods based on the shelf life; and determining and displaying the replenishable quantity of food allowed to be stored in the permitted area based on the available capacity, the number of expired foods, and the number of identified foods. For example, if a 10cm*5cm food item can be placed on a shelf in the refrigerator, and a 5cm*5cm food item is placed on the shelf, the capacity meter installed inside the refrigerator indicates that the available capacity of the food storage area on that shelf is 5cm*5cm, and more food can be placed based on this available capacity. Determining the number of expired foods based on the shelf life can also determine the area on the shelf in the refrigerator where the expired foods are located, and this area can also be used as the permitted storage area for calculating the replenishable quantity of food.

[0050] As an optional embodiment, a notification message is sent to a terminal linked to the refrigerator, including the food identification result. This terminal can be a mobile phone, tablet, computer, or other device. It can instantly receive the notification message including the identification result, allowing the user to immediately know the current status of the food in the refrigerator. Users can also perform other operations based on the identification result, such as automatically reminding them when the food quantity is insufficient or placing an order for new food. For example, a reminder for pre-ordered food quantity can be set; when the pre-ordered food quantity is insufficient, a notification message for replenishment can be sent to the terminal linked to the refrigerator, facilitating timely restocking. Furthermore, the identification result and the terminal linked to the refrigerator can be combined with other applications for greater convenience and a more intelligent and user-friendly experience.

[0051] Figure 2 This is a flowchart of image recognition method two according to an embodiment of the present invention, as follows: Figure 2 As shown, the method includes the following steps:

[0052] Step S202: Receive food images sent by the refrigerator, wherein the food images are acquired by a camera installed inside the refrigerator and include food labels.

[0053] Step S204: Recognize the food image to obtain a first recognition result, wherein the first recognition result includes the result obtained after recognizing the food label in the food image;

[0054] Step S206: The first identification result is sent to the refrigerator, so that the refrigerator can display the food identification result based on the first identification result.

[0055] Through the above steps, the cloud server receives and recognizes food images and sends the first recognition result to the refrigerator, thereby achieving the goal of recognizing food information in the food images. This achieves the technical effect of recognizing food information inside the refrigerator, and solves the technical problems of cumbersome operation and poor user experience when entering food information in related technologies.

[0056] As an optional embodiment, a food image sent by a refrigerator is received. The food image is acquired by a camera installed inside the refrigerator and includes a food label. Recognition is performed based on the food image to ensure that the recognition result is about the food image and that more relevant information about the food can be obtained based on the food label.

[0057] As an optional embodiment, a food image is recognized to obtain a first recognition result. This first recognition result includes the result obtained after recognizing the food label in the food image. Various types of recognition can be performed based on the food image to identify the quantity of food, shelf life, and other information. For example, a quantity recognition model can be used to recognize the quantity of food in the food image. This model is trained using multiple sets of data, including: the image itself, the quantity of food in the image, and the shelf life of the food obtained by recognizing the food label in the food image using Optical Character Recognition (OCR) methods. More detailed features about the food in the food image can be obtained by recognizing other information in the food image using other methods.

[0058] As an optional embodiment, before using a quantity recognition model to identify the quantity of food images and obtain the quantity recognition result, the method includes: acquiring multiple sets of data for training, wherein the multiple sets of training data include: images and the quantity of food in the images; and training an initial model using the multiple sets of training data to obtain a quantity recognition model. Since the multiple sets of training data are all images and the quantity of food in the images, the subsequent use of the trained quantity recognition model to identify the quantity in food images can specifically identify the quantity of food in the images, effectively avoiding problems such as mismatch, inaccuracy, omissions, and repetitions in identification. This allows the food images to learn better, acquiring distinctive features including the quantity of food, and gradually improving the accuracy of identification through continuous training with multiple sets of data.

[0059] As an optional embodiment, the shelf life of food is obtained by recognizing food labels in food images using OCR methods. This includes: recognizing the text on the food labels in the food images using OCR methods to obtain text recognition results; if the text recognition results include the shelf life, matching the production date and shelf life of the food based on the text recognition results; and / or, if the text recognition results do not include the shelf life, obtaining the food category information, and calculating the shelf life of the food based on the category information and the production date matched from the text recognition results. Different processing methods are used in different situations to obtain the shelf life of food. OCR (Optical Character Recognition) technology examines the characters on the food image, confirms the shape of the characters in the image by detecting dark and light patterns, and translates the shape into text using character recognition methods, thereby recognizing the food label.

[0060] Figure 3 This is a flowchart of image recognition method three according to an embodiment of the present invention, as follows: Figure 3 As shown, the method includes the following steps:

[0061] Step S302: The refrigerator uses an internally installed camera to capture food images of the food inside the refrigerator, wherein the food images include food labels.

[0062] In step S304, the refrigerator sends the food image to the cloud server;

[0063] Step S306: The cloud server recognizes the food image, obtains a first recognition result, and returns the first recognition result to the refrigerator. The first recognition result includes the result obtained after recognizing the food label in the food image.

[0064] Step S308: The refrigerator identifies the food image and obtains a second identification result;

[0065] In step S310, the refrigerator determines the food recognition result of the food image based on the first recognition result and the second recognition result, and displays the food recognition result.

[0066] Through the above steps, the refrigerator captures images of food, and the cloud server and the refrigerator respectively recognize the food images to determine the food recognition results. This achieves the goal of recognizing food information in food images, thus achieving the technical effect of recognizing food information inside the refrigerator. This solves the technical problems of cumbersome operation and poor user experience that occur when entering food information in related technologies.

[0067] Based on the above embodiments and optional embodiments, an optional implementation method is provided, which is described in detail below.

[0068] In related technologies, information entry is often done by users themselves. However, this method is cumbersome, has a poor user experience, limits the amount of information that can be entered, and has few expandable functions.

[0069] In view of this, an optional embodiment of the present invention provides a method for managing the shelf life of beverages in a refrigerator. This method employs local static recognition technology combined with OCR (Optical Character Recognition) text recognition. It can not only identify the type and quantity of beverages but also their shelf life information, thus providing shelf-life reminders. Specifically, it can identify beverages through a built-in camera box in the refrigerator, recognizing the beverage type and quantity, and using OCR text matching technology for text recognition. Based on the obtained beverage type and quantity, it can set beverage quantity reminders. Furthermore, based on the identified beverage type and the production date and shelf life recognized by OCR, or the calculated shelf life, it can provide shelf-life reminders.

[0070] Figure 4 This is a design drawing of a refrigerator in an image recognition method according to an optional embodiment of the present invention, such as... Figure 4 As shown, the internal structure design of the refrigerator in an optional embodiment of the present invention is as follows. It should be noted that:

[0071] 1. The camera is located below the upper shelf and is connected to the refrigerator's mainboard via a wire;

[0072] 2. The lower shelf is defined as an identification zone, and the lower bottle holder can be photographed through the main control.

[0073] 3. The main controller uploads the captured image information to the cloud server for recognition, which can realize the recognition of specific beverage types and quantities;

[0074] 4. While recognizing the contents, extract the information from the bottle cap or box lid, perform OCR text recognition again, and match the production date and shelf life according to the recognized format. If only the production date is available and no shelf life is specified, the shelf life can be matched with the product category information recognized in the cloud and calculated to obtain the actual shelf life.

[0075] Figure 5a This is a flowchart of an image recognition method according to an optional embodiment of the present invention, such as... Figure 5a As shown, the following is a detailed description of this optional implementation method:

[0076] S1, actively detects the refrigerator door closing action;

[0077] S2, If the refrigerator door closes, the main controller will take a picture. Figure 6 This is a schematic diagram of an image captured according to an optional embodiment of the present invention, such as... Figure 6 As shown, and uploaded to the cloud server for recognition;

[0078] S3, waiting for the cloud to return the identification information;

[0079] S4, if it receives identification information returned from the cloud. Figure 7 This is a schematic diagram of an optional embodiment of the present invention, which involves OCR recognition after image capture. Figure 7 As shown, information is output based on the identified category, quantity, and OCR-recognized content;

[0080] It should be noted that the operation code can be set as needed during the recognition process. Specifically, in the optional embodiment of the present invention, the code is set as follows:

[0081]

[0082] The S5 records the recognition information in a database locally and displays the recognition information intuitively on the screen using images and text.

[0083] Figure 5b This is a flowchart of an optional embodiment of the extended image recognition method of the present invention, such as... Figure 5b As shown below, the following is an introduction:

[0084] S1, actively detect the predetermined behavior, such as the action of closing the refrigerator door, or specify other actions;

[0085] S2, if the predetermined behavior is detected, the main controller will take a picture;

[0086] S3 will upload the captured images to the cloud server for recognition, and / or perform local recognition;

[0087] S4 is identified using methods such as Optical Character Recognition (OCR), and the identification results include: food type, name, label, quantity, shelf life, and storage environment information;

[0088] S5 will supplement and confirm the first identification result obtained through the cloud server with the second identification result obtained locally to obtain the food identification result;

[0089] S6, based on the food identification results, obtain the available capacity of the food storage area, the number of expired foods, and the number of foods that can be replenished;

[0090] The S7 records the recognition information in a database locally and displays the recognition information intuitively on the screen using images and text.

[0091] The above optional implementation methods can achieve at least the following beneficial effects:

[0092] (1) Automatically recognizes beverage type and quantity information, and can set beverage quantity reminders to facilitate timely replenishment for users;

[0093] (2) By identifying the type of beverage and the production date, shelf life or estimated shelf life recognized by OCR, the shelf life reminder can be realized to avoid beverages being wasted due to expiration;

[0094] (3) Based on the identification results, we can build an ecosystem with beverage companies, such as dairy companies, to provide users with services such as automatic beverage replenishment.

[0095] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0097] Example 2

[0098] According to an embodiment of the present invention, an apparatus for implementing the above-described image recognition method is also provided. Figure 8 This is a structural block diagram of an image recognition device according to an embodiment of the present invention, such as... Figure 8 As shown, the device includes a data acquisition module 802, a first transmitting module 804, a first receiving module 806, and a display module 808. The device will be described in detail below.

[0099] The acquisition module 802 is used to acquire food images of food inside the refrigerator using a camera installed inside the refrigerator, wherein the food images include food labels; the first sending module 804 is connected to the acquisition module 802 and is used to send the food images to a cloud server; the first receiving module 806 is connected to the first sending module 804 and is used to receive a first recognition result returned by the cloud server, wherein the first recognition result includes the result obtained after recognizing the food labels in the food images; the display module 808 is connected to the first receiving module 806 and is used to display the food recognition result based on the first recognition result.

[0100] It should be noted that the above-mentioned acquisition module 802, first transmission module 804, first reception module 806 and display module 808 correspond to steps S102 to S108 in the implementation of the image recognition method one. The multiple modules and the corresponding steps are the same in terms of implementation examples and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0101] Example 3

[0102] According to an embodiment of the present invention, an apparatus for implementing the image recognition method two described above is also provided. Figure 9 This is a structural block diagram of an image recognition device two according to an embodiment of the present invention, such as... Figure 9 As shown, the device includes: a second receiving module 902, an identification module 904, and a second transmitting module 906. The device will be described in detail below.

[0103] The second receiving module 902 is used to receive food images sent by the refrigerator, wherein the food images are captured by a camera installed inside the refrigerator and include food labels; the recognition module 904 is connected to the second receiving module 902 and is used to recognize the food images to obtain a first recognition result, wherein the first recognition result includes the result obtained after recognizing the food labels in the food images; the second sending module 906 is connected to the recognition module 904 and is used to send the first recognition result to the refrigerator, so that the refrigerator can display the food recognition result based on the first recognition result.

[0104] It should be noted that the receiving module 902, the identification module 904 and the second sending module 906 mentioned above correspond to steps S202 to S206 in the second implementation of the image recognition method. The multiple modules and the corresponding steps are the same in terms of implementation examples and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0105] Example 4

[0106] According to an embodiment of the present invention, a system for implementing the above-described image recognition method three is also provided. Figure 10This is a structural block diagram of an image recognition system according to an embodiment of the present invention, such as... Figure 10 As shown, the device includes a refrigerator 1002 and a cloud server 1004. The device will be described in detail below.

[0107] Refrigerator 1002 is used to capture food images of food inside the refrigerator using an internally installed camera, and send the food images, which include food labels, to a cloud server. It is also used to recognize the food images to obtain a second recognition result; and based on the first and second recognition results, to determine and display the food recognition result. Cloud server 1004, connected to refrigerator 1002, is used to recognize the food images, obtain a first recognition result, and return the first recognition result to the refrigerator. The first recognition result includes the result obtained after recognizing the food labels in the food images.

[0108] It should be noted that the refrigerator 1002 and cloud server 1004 mentioned above correspond to the steps in the third method of image recognition. The multiple devices are the same as the corresponding steps in terms of implementation instances and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0109] Example 5

[0110] Embodiments of this disclosure can provide an electronic device, which can be a terminal or a server. In this embodiment, the electronic device, as a terminal, can be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the terminal can also be a mobile terminal or other terminal device.

[0111] Optionally, in this embodiment, the terminal may be located in at least one of a plurality of network devices in a computer network.

[0112] Optionally, Figure 11 This is a structural block diagram of a terminal according to an exemplary embodiment. For example... Figure 11 As shown, the terminal may include: one or more (only one is shown in the figure) processors 111 and a memory 112 for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement any of the above-mentioned image recognition methods.

[0113] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image recognition method and apparatus in this disclosure embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image recognition method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: capturing food images of food inside the refrigerator using a camera installed inside the refrigerator, wherein the food images include food labels; sending the food images to a cloud server; receiving a first recognition result returned by the cloud server, wherein the first recognition result includes the result obtained after recognizing the food labels in the food images; and displaying the food recognition result based on the first recognition result.

[0115] Optionally, the processor may also execute program code for the following steps: displaying food recognition results based on the first recognition result, including: recognizing a food image through the refrigerator's processor to obtain a second recognition result; determining the food recognition result of the food image based on the first and second recognition results, and displaying the food recognition result.

[0116] Optionally, the processor may also execute program code that includes at least one of the following steps: the identification result includes the type of food, the name of the food, the identification of the food, the quantity of the food, the shelf life of the food, and the storage environment information of the food.

[0117] Optionally, the processor may also execute program code for the following steps: when the food identification result includes the quantity of food and the shelf life of food, the method further includes: acquiring the available capacity of the area where food is stored in the refrigerator using a capacity detector installed in the refrigerator; determining the number of expired food items based on the shelf life; determining the replenishable quantity of food allowed to be stored in the area based on the available capacity, the number of expired food items, and the number of identified food items, and displaying the replenishable quantity.

[0118] Optionally, the processor may also execute program code that sends a prompt message to a terminal associated with the refrigerator, wherein the prompt message includes the food identification result.

[0119] Optionally, the processor may also execute program code for the following steps: the food image is an image captured by a camera mounted below a refrigerator shelf.

[0120] In embodiments of this disclosure, the electronic device functions as a server. Figure 12 This is a structural block diagram of a server according to an exemplary embodiment. For example... Figure 12 As shown, the server 120 may include: one or more (only one is shown in the figure) processing components 121, a memory 122 for storing executable instructions of the processing components 121, a power supply component 123 for providing power, a network interface 124 for communicating with an external network, and an I / O input / output interface 125 for data transmission with the outside; wherein, the processing components 121 are configured to execute instructions to implement any of the above-mentioned image recognition methods.

[0121] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image recognition method and apparatus in this disclosure embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image recognition method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The processing component can invoke information and applications stored in the memory through the transmission device to perform the following steps: receiving a food image sent by the refrigerator, wherein the food image is acquired by a camera installed inside the refrigerator and includes a food label; recognizing the food image to obtain a first recognition result, wherein the first recognition result includes the result obtained after recognizing the food label in the food image; and sending the first recognition result to the refrigerator for the refrigerator to display the food recognition result based on the first recognition result.

[0123] Optionally, the above processing component may also execute program code for the following steps: recognizing a food image to obtain a first recognition result, wherein the first recognition result includes the result obtained after recognizing the food label in the food image, including: recognizing the food image through a quantity recognition model to obtain the quantity of food, wherein the quantity recognition model is trained using multiple sets of data, the multiple sets of data including: the image, the quantity of food in the image; recognizing the food label in the food image through an optical character recognition (OCR) method to obtain the shelf life of the food; wherein the first recognition result includes the type of food, the name of the food, the identification of the food, the quantity of the food, the shelf life of the food, and the storage environment information of the food.

[0124] Optionally, the above processing component may also execute program code for the following steps: recognizing food labels in food images using OCR methods to obtain the shelf life of the food, including: recognizing the text of food labels in food images using OCR methods to obtain text recognition results; if the text recognition results include the shelf life, matching the production date and shelf life of the food based on the text recognition results; and / or, if the text recognition results do not include the shelf life, obtaining the category information of the food, and calculating the shelf life of the food based on the category information and the production date matched from the text recognition results.

[0125] Those skilled in the art will understand that Figure 11 , Figure 12 The structure shown is for illustrative purposes only. For example, the terminal mentioned above can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 11 , Figure 12 This does not limit the structure of the aforementioned electronic device. For example, it may also include... Figure 11 , Figure 12 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 11 , Figure 12 The different configurations shown.

[0126] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0127] Example 6

[0128] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, which, when executed by a processor of a terminal, enable the terminal to perform any of the image recognition methods described above. Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0129] Optionally, in this embodiment, the computer-readable storage medium described above can be used to store the program code executed by the image recognition method provided in the above embodiment.

[0130] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0131] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring food images of food inside the refrigerator using a camera installed inside the refrigerator, wherein the food images include food labels; sending the food images to a cloud server; receiving a first recognition result returned by the cloud server, wherein the first recognition result includes the result obtained after recognizing the food labels in the food images; and displaying the food recognition result based on the first recognition result.

[0132] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: displaying food recognition results based on a first recognition result, including: recognizing a food image through the refrigerator's processor to obtain a second recognition result; determining the food recognition result of the food image based on the first and second recognition results, and displaying the food recognition result.

[0133] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: the identification result includes at least one of the following: the type of food, the name of the food, the identifier of the food, the quantity of the food, the shelf life of the food, and the storage environment information of the food.

[0134] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: when the food identification result includes the quantity of food and the shelf life of the food, the method further includes: acquiring the available capacity of the area in the refrigerator where food is stored using a capacity detector installed in the refrigerator; determining the number of expired food items based on the shelf life; determining the number of food items that can be replenished in the area based on the available capacity, the number of expired food items, and the number of identified food items, and displaying the number of food items that can be replenished.

[0135] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: the method further includes sending a prompt message to a terminal bound to the refrigerator, wherein the prompt message includes food identification results.

[0136] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: the food image is an image captured by mounting a camera under the refrigerator shelf.

[0137] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: receiving a food image sent by a refrigerator, wherein the food image is acquired by a camera installed inside the refrigerator and includes a food label; recognizing the food image to obtain a first recognition result, wherein the first recognition result includes the result obtained after recognizing the food label in the food image; and sending the first recognition result to the refrigerator for the refrigerator to display the food recognition result based on the first recognition result.

[0138] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: recognizing a food image to obtain a first recognition result, including: recognizing the food image using a quantity recognition model to obtain the quantity of food, wherein the quantity recognition model is trained using multiple sets of data, the multiple sets of data including: an image, the quantity of food in the image; recognizing a food label in the food image using an optical character recognition (OCR) method to obtain the shelf life of the food; wherein the first recognition result includes the type of food, the name of the food, the identification of the food, the quantity of the food, the shelf life of the food, and the storage environment information of the food.

[0139] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: recognizing a food label in a food image using an OCR method to obtain the shelf life of the food, including: recognizing the text of the food label in the food image using an OCR method to obtain a text recognition result; if the text recognition result includes a shelf life, matching the production date and shelf life of the food based on the text recognition result; and / or, if the text recognition result does not include a shelf life, obtaining the category information of the food, and calculating the shelf life of the food based on the category information and the production date matched from the text recognition result.

[0140] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: the refrigerator acquires food images of food inside the refrigerator through an internally installed camera, wherein the food images include food labels; the refrigerator sends the food images to a cloud server; the cloud server identifies the food images, obtains a first identification result, and returns the first identification result to the refrigerator, wherein the first identification result includes the result obtained after identifying the food labels in the food images; the refrigerator identifies the food images, obtains a second identification result; based on the first identification result and the second identification result, the refrigerator determines the food identification result of the food images and displays the food identification result.

[0141] In an exemplary embodiment, a computer program product is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform any of the image recognition methods described above.

[0142] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0143] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0147] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0148] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image recognition method, characterized in that, include: A camera installed inside the refrigerator captures food images of the food inside the refrigerator based on the food characteristics of the food inside the refrigerator, wherein the food images include food labels; Send the food image to the cloud server; The system receives a first recognition result returned by the cloud server, wherein the first recognition result includes the result obtained after recognizing the food label in the food image; Based on the first identification result, the food identification result is displayed; the food identification result includes: the quantity of the food and the shelf life of the food; The method further includes: collecting the available capacity of the area in the refrigerator where the food is stored using a capacity detector installed in the refrigerator; determining the number of expired food items based on the shelf life; determining the number of food items that can be replenished in the area based on the available capacity, the number of expired food items, and the number of identified food items, and displaying the number of food items that can be replenished.

2. The method according to claim 1, characterized in that, Based on the first identification result, the food identification result is displayed, including: The refrigerator's processor identifies the food image to obtain a second identification result; Based on the first recognition result and the second recognition result, the food recognition result of the food image is determined and displayed.

3. The method according to claim 1, characterized in that, The food identification results also include at least one of the following: The type of food, the name of the food, the identification of the food, and the storage environment information of the food.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: A prompt message is sent to the terminal bound to the refrigerator, wherein the prompt message includes the food identification result.

5. The method according to claim 4, characterized in that, The food image is an image captured by installing the camera below the refrigerator shelf.

6. An image recognition method, characterized in that, include: The refrigerator receives food images sent by the refrigerator. The food images are acquired by a camera installed inside the refrigerator based on the food characteristics of the food inside the refrigerator. The food images include food labels. The food image is identified to obtain a first identification result, wherein the first identification result includes the result obtained after identifying the food label in the food image; The first identification result is sent to the refrigerator, which then displays the food identification result based on the first identification result. Wherein, if the food identification result includes the quantity of the food and its shelf life, the refrigerator also displays the replenishable quantity. The replenishable quantity is determined by a capacity sensor installed inside the refrigerator, which collects the available capacity of the area where the food is stored. Based on the shelf life, the expired quantity of expired food is determined. Based on the available capacity, the expired quantity, and the identified quantity of food, the allowed quantity of food to be stored in the area is determined.

7. The method according to claim 6, characterized in that, The food image is identified to obtain a first identification result, including: The quantity recognition model is used to identify the quantity of food in the food image. The quantity recognition model is trained using multiple sets of data, including: an image and the quantity of food in the image. The shelf life of the food is obtained by recognizing the food label in the food image using the Optical Character Recognition (OCR) method. The first identification result includes the type of food, the name of the food, the identifier of the food, the quantity of the food, the shelf life of the food, and the storage environment information of the food.

8. The method according to claim 7, characterized in that, The step of identifying the food label in the food image using the OCR method to obtain the shelf life of the food includes: The text on the food label in the food image is recognized using the OCR method to obtain the text recognition result; If the text recognition result includes a shelf life, the production date and shelf life of the food are matched based on the text recognition result; and / or, if the text recognition result does not include the shelf life, the category information of the food is obtained, and the shelf life of the food is calculated based on the category information and the production date matched from the text recognition result.

9. An image recognition method, characterized in that, include: The refrigerator uses an internal camera to capture images of the food inside the refrigerator based on the food characteristics, and the food images include food labels. The refrigerator sends the food image to a cloud server; The cloud server identifies the food image, obtains a first identification result, and returns the first identification result to the refrigerator. The first identification result includes the result obtained after identifying the food label in the food image. The refrigerator identifies the food image and obtains a second identification result; The refrigerator determines the food recognition result of the food image based on the first recognition result and the second recognition result, and displays the food recognition result; the food recognition result includes: the quantity of the food and the shelf life of the food; The method further includes: The system uses a capacity sensor installed inside the refrigerator to collect the available capacity of the area where the food is stored. Based on the shelf life, it determines the number of expired food items. Based on the available capacity, the number of expired food items, and the number of identified food items, it determines the number of food items that can be replenished and displays the number of food items that can be replenished.

10. An image recognition device, characterized in that, include: The acquisition module is used to acquire food images of the food inside the refrigerator based on the food characteristics of the food inside the refrigerator through a camera installed inside the refrigerator, wherein the food images include food labels; The first sending module is used to send the food image to a cloud server; The first receiving module is used to receive the first recognition result returned by the cloud server, wherein the first recognition result includes the result obtained after recognizing the food label in the food image; The display module is used to display the food identification result based on the first identification result; the food identification result includes: the quantity of the food and the shelf life of the food; and is also used to collect the available capacity of the area in the refrigerator where the food is stored by means of a capacity detector installed in the refrigerator; determine the number of expired food items based on the shelf life; determine the number of food items that can be replenished in the area based on the available capacity, the number of expired food items, and the number of identified food items, and display the number of food items that can be replenished.

11. An image recognition device, characterized in that, include: The second receiving module is used to receive food images sent by the refrigerator. The food images are obtained by a camera installed inside the refrigerator based on the food characteristics of the food inside the refrigerator. The food images include food labels. The recognition module is used to recognize the food image and obtain a first recognition result, wherein the first recognition result includes the result obtained after recognizing the food label in the food image; The second sending module is used to send the first identification result to the refrigerator, and the refrigerator displays the food identification result based on the first identification result; wherein, when the food identification result includes: the quantity of the food and the shelf life of the food, the refrigerator is also used to display the replenishable quantity; the replenishable quantity is collected by a capacity detector installed in the refrigerator to collect the available capacity of the area of ​​the refrigerator where the food is stored; based on the shelf life, the expired quantity of expired food is determined; based on the available capacity, the expired quantity, and the quantity of identified food, the allowed quantity of food to be stored in the area is determined.

12. An image recognition system, characterized in that, include: Refrigerators and cloud servers, among which, The refrigerator is used to capture food images of the food inside the refrigerator based on the food characteristics of the food inside the refrigerator through a camera installed inside the refrigerator, and to send the food images to a cloud server, wherein the food images include food labels; The cloud server is used to recognize the food image, obtain a first recognition result, and return the first recognition result to the refrigerator. The first recognition result includes the result obtained after recognizing the food label in the food image. The refrigerator is further configured to recognize the food image to obtain a second recognition result; and based on the first recognition result and the second recognition result, determine the food recognition result of the food image and display the food recognition result; the food recognition result includes: the quantity of the food and the shelf life of the food; and is further configured to collect the available capacity of the area in the refrigerator where the food is stored by means of a capacity sensor installed in the refrigerator; based on the shelf life, determine the number of expired food items; based on the available capacity, the number of expired food items, and the number of recognized food items, determine the number of food items that can be replenished and stored in the area, and display the number of food items that can be replenished.

Images