Information management method and device, medium and terminal equipment
By setting up a camera and weight sensor in the refrigerator, automatically identifying and recording item information, the problem of low entry efficiency of existing refrigerator systems is solved, automatic identification and update of item names, quantities and locations is realized, and the management of items in the refrigerator is improved practicality.
Patent Information
- Application Number
- CN202410941815.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-15
AI Technical Summary
The existing refrigerator system is inefficient when entering items, especially when buying more items, it requires frequent manual entry, and its functions are low, making it impossible to automatically identify the number and location of items.
By setting up a camera in the refrigerator, identifying and obtaining item images, using pre-trained information extraction models to identify item names and quantities, and combining weight sensors to obtain item weight and position information, automatically binding and updating item information, providing voice and display interface Prompt the user for correction.
It improves the efficiency of item information entry, and users can view specific information about items in the refrigerator at any time, enhancing functionality and practicality, and reducing the burden of manual entry.
Smart Images

Figure CN118968501A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of electronic communication technology, in particular to the field of information management technology, and more particularly to an information management method, apparatus, medium and terminal equipment. Background Art
[0002] The invention of the refrigerator allows people to buy enough food for a few days and store it in the refrigerator for freshness and refrigeration, without having to worry about the food going bad because it cannot be eaten temporarily.
[0003] In order to facilitate people to obtain information about the current items stored at any time without opening the refrigerator, such as type and quantity, it is now possible to connect the refrigerator system to a mobile phone APP, so that people can view the information of the items stored inside the refrigerator through their mobile phones. Before realizing the remote viewing of the information of the items stored inside the refrigerator through a mobile phone, people need to manually select the type and quantity of items when putting items into the refrigerator to complete the information entry operation of the items. However, when there are many items to be purchased, people need to manually enter the item information frequently, which affects efficiency. At the same time, most refrigerators on the market can only realize the function of viewing the type of information of items, which is less practical. Summary of the invention
[0004] The embodiments of the present application provide an information management method, device, medium and terminal device. The information management method provided by the embodiments of the present application can improve the efficiency of item information entry and allow users to understand the status of items in the refrigerator in many aspects, which is more functional and practical.
[0005] The present application provides an information management method, which is applied to a refrigerator and includes:
[0006] When an object placed in the refrigerator is identified, controlling the camera to obtain an image of the object;
[0007] Inputting the object image into a pre-trained information extraction model to perform an information extraction operation to obtain the object name and the object quantity contained in the object image;
[0008] When detecting that the load-bearing weight of an item storage platform located in the refrigerator changes, obtaining the position information of the item storage platform and the item weight of the item;
[0009] The item name is bound to the corresponding item quantity, location information and item weight, or the total quantity, location summary information and total item weight of the item name to which the item name is bound are updated.
[0010] In the information management method described in the embodiment of the present application, after the controlling camera acquires the object image of the object, the method further includes:
[0011] Providing a selection entry interface and generating a first voice prompt message to prompt the user to select whether the type of the item to be put in this time is a newly added type on the selection entry interface;
[0012] If it is a newly added type, the time when the item is placed is recorded, and the placement time is bound to the item name corresponding to the item.
[0013] In the information management method described in the embodiment of the present application, after prompting the user to select on the operation interface whether the type of the item to be placed this time is a newly added type, the method further includes:
[0014] If the user selects the type of the item to be placed this time as a non-newly added type on the selection and entry interface, all historical placement times of the item are retrieved for the user to view.
[0015] In the information management method described in the embodiment of the present application, after binding the item name with the corresponding item quantity, location information and item weight, or updating the total quantity of items, location summary information and total weight of items to which the item name is bound, the method further includes:
[0016] Generate a second voice prompt message, or provide a display interface to display the first text prompt message, to prompt the user one or more of the name of the item, the number of items, and the location of the item currently placed;
[0017] A user correction interface is provided for the user to determine whether to correct one or more of the item name, item quantity and item position of the item on the user correction interface according to the second voice prompt information or the first text prompt information.
[0018] In the information management method described in the embodiment of the present application, after providing the user correction interface, the method further includes:
[0019] In response to a first information entry operation on the user correction interface, determining whether to modify one or more of the item name, item quantity, and item location according to the first information entry operation;
[0020] In response to a second information entry operation on the user correction interface, one or more of the item name, item quantity, and item location are modified according to the second information entry operation.
[0021] In the information management method described in the embodiment of the present application, after binding the item name with the corresponding item quantity, location information and item weight, or updating the total quantity of items, location summary information and total weight of items to which the item name is bound, the method further includes:
[0022] When a change in the load-bearing weight of an item storage platform is detected, obtaining the position information of the item storage platform, the weight of the items, and the change value of the current load-bearing weight of the item storage platform;
[0023] Determine the status of the items currently carried by the item storage platform according to the change value, wherein the item status includes placement and removal;
[0024] If the item status is taken out, the camera is controlled to move to the position corresponding to the position information according to the position information, and the name and quantity of the items currently carried by the item storage platform are obtained based on the camera;
[0025] Generate a third voice prompt message, or provide a display interface to display the second text prompt message, to prompt the user one or more of the item name, item quantity and item location of the item currently taken out.
[0026] In the information management method described in the embodiment of the present application, after binding the item name with the corresponding item quantity, location information and item weight, or updating the total quantity of items, location summary information and total weight of items to which the item name is bound, the method further includes:
[0027] receiving an information viewing request, and determining the name of an object to be viewed and a viewing angle according to the information viewing request;
[0028] According to the item name and the viewing angle, the item quantity, item location, item image and placement time corresponding to the item name are obtained and sent to the sender of the information viewing request.
[0029] Correspondingly, another aspect of the embodiment of the present application further provides an information management device, which is applied to a refrigerator. The information management device includes:
[0030] A recognition module, used for controlling a camera to obtain an image of an object placed in the refrigerator when the object is recognized;
[0031] An extraction module, used for inputting the object image into a pre-trained information extraction model to perform an information extraction operation, and obtaining the object name and the object quantity contained in the object image;
[0032] a determination module, configured to obtain position information of an item storage platform and an item weight of an item when detecting that a load-bearing weight of an item storage platform located in the refrigerator changes;
[0033] The binding module is used to bind the item name with its corresponding item quantity, location information and item weight, or to update the total quantity, location summary information and total weight of the items to which the item name is bound.
[0034] Correspondingly, another aspect of the embodiment of the present application further provides a storage medium, wherein the storage medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute the information management method as described above.
[0035] Correspondingly, on the other hand, an embodiment of the present application further provides a terminal device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the information management method as described above.
[0036] The embodiment of the present application provides an information management method, device, medium and terminal device. The method controls the camera to obtain the item image of the item when the item placed in the refrigerator is identified; inputs the item image into the pre-trained information extraction model to perform information extraction operation, and obtains the item name and item quantity contained in the item image; when the load weight of an item storage platform located in the refrigerator is detected to change, the location information of the item storage platform is determined; and the item name and its corresponding item quantity and location information are bound. Using the information management method provided by the embodiment of the present application, when the item is placed in the refrigerator, the camera can be used to extract information from the item to quickly identify the item name and item quantity of the current item, and after the item is stored in the item storage platform of the refrigerator, the location information and item weight of the current item can be identified, so that the user can understand the situation of the items in the refrigerator in many aspects. After obtaining multiple user-required information of the current item, the system can automatically bind the item name and its corresponding item quantity, location information, and item weight, or update the total number of items, location summary information, and total weight of items to which the item name has been bound, so that the user can view the specific information of the items in the refrigerator at any time, and the functionality and practicality are stronger. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.
[0038] Figure 1A flowchart of the information management method provided in an embodiment of the present application.
[0039] Figure 2 Schematic diagram of the internal structure of a refrigerator.
[0040] Figure 3 This is a schematic diagram of the structure of the intelligent food management terminal system provided in an embodiment of the present application.
[0041] Figure 4 A schematic diagram of the structure of an information management device provided in an embodiment of the present application.
[0042] Figure 5 Another structural schematic diagram of the information management device provided in an embodiment of the present application.
[0043] Figure 6 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0045] The embodiment of the present application provides an information management method, which can perform information extraction operations when placing items in a refrigerator to identify the item name and item quantity of the current item, and can identify the location information and item weight of the current item after the item is stored in the item storage platform of the refrigerator. After obtaining multiple pieces of information required by the user for the current item, the system can automatically bind the item name and its corresponding item quantity, location information, and item weight, or update the total number of items, location summary information, and total item weight to which the item name has been bound, so that the user can view the specific information of the items in the refrigerator at any time.
[0046] The term "and / or" in this application can be a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.
[0047] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to the steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present application.
[0048] The information management method involved in the embodiments of the present application is mainly applied to terminal devices, such as refrigerators, freezers, fresh-keeping cabinets, etc. In addition to terminal devices, it can also be applied to other terminal devices that can control the operation of terminal devices, such as tablet computers, laptops, desktop computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and servers, etc., which are not limited here. Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms and other basic cloud computing services, which are not limited here.
[0049] For ease of understanding, the specific process in the embodiments of the present application is described below. Figure 1 . Figure 1A flowchart of an embodiment of an information management method provided in an embodiment of the present application.
[0050] exist Figure 1 In the illustrated embodiment, the method may include the following steps:
[0051] Step 101, when an object placed in the refrigerator is identified, control the camera to obtain an image of the object.
[0052] In this solution, users can put items such as food or any items that need to be kept fresh and refrigerated into the refrigerator. The system will identify and manage the information of the items (such as the quantity, location information, and weight of the items) for users to view on mobile terminals such as mobile phones or display screens set on the refrigerator.
[0053] In this embodiment, a camera is set inside the refrigerator to obtain an image of an item, and subsequent information extraction operations are performed based on the obtained image of the item. At the same time, a button is set on the refrigerator for the user to select. Pressing the button can control the refrigerator to enter the storage mode. In the storage mode, the camera will automatically move to a specified position or stay at a default fixed position and start the image shooting function. At this time, the user needs to move the item to the position of the camera so that the camera can accurately obtain the image of the item.
[0054] It should be noted that the camera is preferably a miniature camera to avoid collision with items inside the refrigerator during movement, and the track that provides guidance for the camera is preferably set on the inner wall of the refrigerator, so that the camera can move to the specified position along the track set on the inner wall of the refrigerator after receiving the command. The track structure of the camera is not specifically limited.
[0055] Step 102: Input the object image into a pre-trained information extraction model to perform an information extraction operation to obtain the object name and the object quantity contained in the object image.
[0056] In this embodiment, after the system performs information extraction operations on the acquired object image through the built-in information extraction model, the object name and item quantity can be identified. For example, according to the recognition result, it can be known that the item name of the current object is apple and the item quantity is 2.
[0057] It should be noted that the information extraction model can be trained based on decision trees, gradient boosting trees, convolutional neural networks, naive Bayes and logistic regression models. Taking the convolutional neural network model as an example, the training process of the information extraction model is explained, including:
[0058] A training sample of the information extraction model to be trained is obtained, wherein the training sample includes an image with a label; a feature extraction is performed on the image in the training sample by the information extraction model to be trained to obtain an image feature vector corresponding to the image; the name and quantity of the object in the image in the training sample are extracted based on the image feature vector by the information extraction model to be trained to obtain an image recognition result; based on the recognition result and the label of the image, the parameters of the information extraction model to be trained are adjusted to obtain a pre-trained information extraction model.
[0059] Step 103, when it is detected that the load-bearing weight of an item storage platform located in the refrigerator changes, the position information of the item storage platform and the item weight of the item are obtained.
[0060] In this embodiment, if Figure 2 As shown, the item storage platform / weighable platform is used to place items, and its specific structure can be either a storage box or a storage plate, which is not limited here. However, in order to facilitate the classification of items, it is preferred to set a separate item storage platform for each type of item, and a label indicating the name of the item can also be set on each item storage platform.
[0061] In this embodiment, the item storage platform is not only used to place items, but also to identify the location information and item weight of the items, which are also information that users care about. In order to realize that the item storage platform has the ability to identify the location information and item weight, it is necessary to set a weight acquisition device such as a weight sensor on the item storage platform to sense the weight signal corresponding to the item weight, and after conversion by a processor for receiving the weight signal, a specific value can be obtained to determine the item weight, and the processor sends the item weight to the system to inform the system of the current item weight. The location information of the item is mainly determined by the position of the item storage platform itself where the item is placed. Specifically, the location information of each item storage platform (such as serial number 1) can be entered into the system in advance, and the processor for receiving the weight signal sent by the weight sensor is configured to "obtain the location information of the current item storage platform when the load-bearing weight at any item storage platform is detected to change", and send the location information to the system to inform the system of the current item location information.
[0062] Step 104: Bind the item name with its corresponding item quantity, location information and item weight, or update the total item quantity, location summary information and total item weight to which the item name is bound.
[0063] It should be explained that the total number of items refers to the total number of items in the refrigerator, the location summary information refers to all location information of an item in the refrigerator, and the total weight of items refers to the total weight of an item in the refrigerator.
[0064] In this embodiment, the items put into the refrigerator can be divided into two categories: one is newly added items, that is, the information of such items has not been entered into the refrigerator database, indicating that such items do not exist in the refrigerator at this time. The other is the information of such items that have been entered into the refrigerator, indicating that such items may still exist in the refrigerator at this time.
[0065] In the first case, if the system detects that the type of item currently placed in the refrigerator is a newly added type, the system will automatically bind the item name with the corresponding item quantity, location information and item weight, so that the user can view the item information through a mobile terminal such as a mobile phone or a display screen set on the refrigerator. In the second case, if the system detects that the type of item currently placed in the refrigerator is not a newly added type, the system will automatically update the total number of items, location summary information and total weight of items bound to the item name, so that the user can know the current inventory of a certain type of item in the refrigerator, avoiding the user from continuing to purchase such items without knowing that there is a large amount of inventory in the refrigerator, causing waste.
[0066] It should be noted that in order to allow the system to know whether the type of items currently placed in the refrigerator is a newly added type, in some embodiments, after the user moves the item to the location of the camera to perform an information extraction operation, a user selection and entry interface can be provided, and a first voice prompt message can be generated to prompt the user to select whether the type of item placed this time is a newly added type on the selection and entry interface; and determine in turn whether the type of item currently placed in the refrigerator is a newly added type.
[0067] In some embodiments, if it is determined that the type of item currently placed in the refrigerator is a newly added type, the time when the item was placed can also be recorded, and the placement time can be bound to the item name corresponding to the item, so that the user can know the time when the item was placed while viewing the item information, which is convenient for the user to use the item in time within the insurance period, and can also deal with the item in time when it is stored for a long time.
[0068] In some embodiments, if the user selects that the type of item to be placed this time is not a new type on the selection and entry interface, all historical placement times of the item can be retrieved for the user to view, reminding the user of the inventory status of the items currently placed in the refrigerator. After obtaining all historical placement times of the items, the user can consider whether to select items that have been stored for a longer time for replacement.
[0069] In some embodiments, after binding the item name with its corresponding item quantity, location information and item weight, or updating the total quantity of items, location summary information and total weight of items to which the item name is bound, a second voice prompt message can also be generated, or a display interface can be provided to display the first text prompt message, so as to prompt the user of one or more of the item name, item quantity and item location of the item currently placed.
[0070] In some embodiments, after the second voice prompt information or the first text prompt information is displayed, a user correction interface may be provided, so that the user can determine whether to correct one or more of the item name, item quantity, and item location of the item on the user correction interface according to the second voice prompt information or the first text prompt information, so as to avoid the system from misidentifying the item information. If the user determines that there is no error, the subsequent information binding or information update operations will be continued. If the user determines that there is an error, the user can actively change the misidentified information through the user correction interface.
[0071] In some embodiments, in order to facilitate some users with poor object recognition ability, such as visually impaired users, to take out the items they want from the refrigerator, one or more of the item name, item quantity and item location may be announced or displayed to the user when the user takes out the item.
[0072] In order to enable the system to identify the item status of the current item, that is, putting in or taking out, when it is detected that the load-bearing weight of an item storage platform has changed, the position information and item weight of the item storage platform and the change value of the current load-bearing weight of the item storage platform can be determined; the item status currently carried by the item storage platform is determined according to the change value (if the change value is an increase, the item status is putting in, and if the change value is a decrease, the item status is taking out); if the item status is taking out, the camera is controlled to move to the position corresponding to the position information according to the position information, and the item name and item quantity of the item currently carried by the item storage platform are obtained based on the camera; a third voice prompt message is generated, or a display interface is provided to display the second text prompt message, so as to prompt the user to select one or more of the item name, item quantity and item position of the item currently taken out.
[0073] In some embodiments, the user can send an information viewing request to the refrigerator through a mobile terminal such as a mobile phone or a display screen set on the refrigerator. After receiving the request, the system determines the name of the item to be viewed and the viewing angle according to the information viewing request; determines the location of the item according to the item name and the viewing angle, and adjusts the angle of the camera according to the viewing angle to photograph the item to obtain the item image, and then obtains the item quantity, item location, item image and placement time corresponding to the item name and sends them to the sender of the corresponding information viewing request.
[0074] In some embodiments, when it is detected that the refrigerator door is in a closed state and the change in the current load-bearing weight of the item storage platform is displayed as a decrease, an alarm may be issued to inform the user that there may be a problem of items collapsing in the refrigerator.
[0075] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0076] In specific implementation, the present application is not limited by the execution order of the various steps described. If no conflict occurs, some steps can be performed in other orders or simultaneously.
[0077] As can be seen from the above, the information management method provided by the embodiment of the present application, when the object placed in the refrigerator is identified, the camera is controlled to obtain the object image of the object; the object image is input into the pre-trained information extraction model for information extraction operation, and the object name and the number of objects contained in the object image are obtained; when the load-bearing weight of an object storage platform located in the refrigerator is detected to change, the position information of the object storage platform and the object weight of the object are obtained; the object name is bound to the corresponding number of objects, position information and object weight, or the total number of objects, position summary information and total weight of objects bound to the object name are updated. Using the information management method provided by the embodiment of the present application, the information extraction operation can be performed when the object is placed in the refrigerator to identify the object name and the number of objects of the current object, and the position information and the object weight of the current object can be identified after the object is stored in the object storage platform of the refrigerator. After obtaining multiple user-required information of the current object, the system can automatically bind the object name and its corresponding number of objects, position information, and object weight, or update the total number of objects, position summary information and total weight of objects bound to the object name, so that the user can view the specific information of the objects in the refrigerator at any time.
[0078] The embodiment of the present application also provides an intelligent food management terminal system, which can be integrated into a terminal device.
[0079] See also Figure 3 , Figure 3 A schematic diagram of the structure of the food intelligent management terminal system provided in the embodiment of the present application. The food intelligent management terminal system includes a data processing module for identifying the type and quantity of food, the weight and position of food on the storage plate, and data anomaly correction; a camera acquisition device for collecting object images of food; a weight acquisition device for obtaining the weight of the storage plate; a data display and user interaction module for data interaction with users; and a local data storage module for storing data. The user can view the food information inside the refrigerator through the cloud on the mobile phone APP.
[0080] The embodiment of the present application also provides an information management device, which can be integrated in a terminal device.
[0081] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of the information management device provided in the embodiment of the present application. The information management device 30 may include:
[0082] The recognition module 31 is used to control the camera to obtain an image of the object when an object placed in the refrigerator is recognized;
[0083] An extraction module 32 is used to input the object image into a pre-trained information extraction model to perform an information extraction operation to obtain the object name and the object quantity contained in the object image;
[0084] A determination module 33, configured to obtain the position information of an item storage platform and the item weight of the item when detecting that the load weight of an item storage platform located in the refrigerator changes;
[0085] The binding module 34 is used to bind the item name with its corresponding item quantity, location information and item weight, or to update the total quantity, location summary information and total weight of the items to which the item name is bound.
[0086] In some embodiments, the device also includes a first prompt module, which is used to provide a selection and entry interface and generate a first voice prompt message, prompting the user to select on the selection and entry interface whether the type of the item being placed this time is a new type; if it is a new type, the time when the item is placed is recorded, and the placement time is bound to the item name corresponding to the item.
[0087] In some embodiments, the device further includes a retrieval module for retrieving all historical placement times of the item for the user to view if the user selects a non-newly added type for the item to be placed on the selection and entry interface.
[0088] In some embodiments, the device also includes a second prompt module for generating a second voice prompt message, or providing a display interface to display the first text prompt message, so as to prompt the user of one or more of the item name, item quantity and item location of the item currently placed.
[0089] In some embodiments, the device also includes a third prompt module, which is used to obtain the position information of the item storage platform, the item weight of the item, and the change value of the current load-bearing weight of the item storage platform when a change in the load-bearing weight of an item storage platform is detected; determine the status of the items currently carried by the item storage platform according to the change value, and the item status includes putting in and taking out; if the item status is taking out, control the camera to move to the position corresponding to the position information according to the position information, and obtain the item name and item quantity of the items currently carried by the item storage platform based on the camera; generate a third voice prompt message, or provide a display interface to display the second text prompt message, so as to prompt the user to select one or more of the item name, item quantity and item position of the currently taken out item.
[0090] In some embodiments, the device also includes a modification module for providing a user correction interface for the user to determine whether to correct one or more of the item name, item quantity and item location of the item on the user correction interface based on the second voice prompt information or the first text prompt information.
[0091] In some embodiments, the device also includes a receiving module for receiving an information viewing request, and determining the name of the item to be viewed and the viewing angle according to the information viewing request; based on the item name and the viewing angle, obtaining the item quantity, item location, item image and placement time corresponding to the item name and sending them to the sender of the information viewing request.
[0092] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same entity or several entities.
[0093] As can be seen from the above, the information management device 30 provided in the embodiment of the present application, wherein the identification module 31 is used to control the camera to obtain the item image of the item when the item placed in the refrigerator is identified; the extraction module 32 is used to input the item image into a pre-trained information extraction model to perform an information extraction operation, and obtain the item name and item quantity of the item contained in the item image; the determination module 33 is used to obtain the position information of an item storage platform located in the refrigerator and the item weight of the item when it is detected that the load-bearing weight of the item storage platform located in the refrigerator has changed; the binding module 34 is used to bind the item name with its corresponding item quantity, location information and item weight, or to update the total quantity of items, location summary information and total weight of items to which the item name has been bound.
[0094] See also Figure 5 , Figure 5Another structural diagram of the information management device provided in the embodiment of the present application, the information management device 30 includes a memory 120, one or more processors 180, and one or more applications, wherein the one or more applications are stored in the memory 120 and configured to be executed by the processor 180; the processor 180 may include an identification module 31, an extraction module 32, a determination module 33 and a binding module 34. For example, the structure and connection relationship of the above components may be as follows:
[0095] The memory 120 can be used to store applications and data. The applications stored in the memory 120 include executable codes. The applications can be composed of various functional modules. The processor 180 executes various functional applications and data processing by running the applications stored in the memory 120. In addition, the memory 120 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 120 may also include a memory controller to provide the processor 180 with access to the memory 120.
[0096] The processor 180 is the control center of the device, and uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the device and processes data by running or executing applications stored in the memory 120, and calling data stored in the memory 120, so as to monitor the device as a whole. Optionally, the processor 180 may include one or more processing cores; preferably, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs.
[0097] Specifically in this embodiment, the processor 180 will load the executable code corresponding to the process of one or more applications into the memory 120 according to the following instructions, and the processor 180 will run the application stored in the memory 120 to implement various functions:
[0098] A recognition instruction, used to control the camera to obtain an image of the object when an object placed in the refrigerator is recognized;
[0099] An extraction instruction, used to input the object image into a pre-trained information extraction model to perform an information extraction operation, and obtain the object name and the object quantity contained in the object image;
[0100] A determination instruction is used to obtain the position information of an item storage platform and the item weight of the item when a change in the load weight of the item storage platform located in the refrigerator is detected;
[0101] The binding instruction is used to bind the item name with its corresponding item quantity, location information and item weight, or to update the total quantity, location summary information and total weight of the items to which the item name is bound.
[0102] In some embodiments, the program also includes a first prompt instruction, which is used to provide a selection and entry interface and generate a first voice prompt message, prompting the user to select on the selection and entry interface whether the type of item being placed this time is a new type; if it is a new type, the time when the item is placed is recorded, and the placement time is bound to the item name corresponding to the item.
[0103] In some embodiments, the program further includes a retrieval instruction for retrieving all historical insertion times of the item for the user to view if the user selects the type of item to be inserted this time as a non-newly added type on the selection and entry interface.
[0104] In some embodiments, the program also includes a second prompt instruction for generating a second voice prompt message, or providing a display interface to display the first text prompt message to prompt the user one or more of the item name, item quantity and item location of the item currently placed.
[0105] In some embodiments, the program also includes a third prompt instruction, which is used to obtain the position information of the item storage platform, the item weight of the item, and the change value of the current load-bearing weight of the item storage platform when a change in the load-bearing weight of an item storage platform is detected; determine the status of the items currently carried by the item storage platform according to the change value, and the item status includes putting in and taking out; if the item status is taking out, control the camera to move to the position corresponding to the position information according to the position information, and obtain the item name and item quantity of the items currently carried by the item storage platform based on the camera; generate a third voice prompt message, or provide a display interface to display the second text prompt message, so as to prompt the user to select one or more of the item name, item quantity and item position of the currently taken out item.
[0106] In some embodiments, the program also includes modification instructions for providing a user correction interface for the user to determine whether to correct one or more of the item name, item quantity and item location of the item on the user correction interface based on the second voice prompt information or the first text prompt information.
[0107] In some embodiments, the program also includes a receiving instruction for receiving an information viewing request, determining the name of the item to be viewed and the viewing angle based on the information viewing request; based on the item name and the viewing angle, obtaining the item quantity, item location, item image and placement time corresponding to the item name and sending them to the sender of the information viewing request.
[0108] An embodiment of the present application also provides a terminal device.
[0109] See also Figure 6 , Figure 6 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application is shown, and the terminal device can be used to implement the information management method provided in the above embodiment.
[0110] like Figure 6 As shown, the terminal device 1200 may include an RF (Radio Frequency) circuit 110, a memory 120 including one or more (only one is shown in the figure) computer-readable storage media, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a transmission module 170, a processor 180 including one or more (only one is shown in the figure) processing cores, and a power supply 190. Those skilled in the art will understand that Figure 6 The structure of the terminal device 1200 shown in the figure does not constitute a limitation on the terminal device 1200, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0111] The RF circuit 110 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 110 may include various existing circuit elements for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a user identity module (SIM) card, a memory, etc. The RF circuit 110 can communicate with various networks such as the Internet, an intranet, a wireless network, or communicate with other devices through a wireless network.
[0112] The memory 120 can be used to store software programs and modules, such as the program instructions / modules corresponding to the information management method in the above embodiment. The processor 180 executes various functional applications and data processing by running the software programs and modules stored in the memory 120. The vibration reminder mode can be automatically selected according to the current scene of the terminal device to perform information management, which can not only ensure that scenes such as meetings are not disturbed, but also ensure that users can perceive incoming calls, thereby improving the intelligence of the terminal device. The memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 120 may further include a memory remotely arranged relative to the processor 180, and these remote memories may be connected to the terminal device 1200 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0113] The input unit 130 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 130 may include a touch-sensitive surface 131 and other input devices 132. The touch-sensitive surface 131, also known as a touch display screen or a touch pad, can collect the user's touch operations on or near it (such as the user's operation on or near the touch-sensitive surface 131 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 131 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 180, and can receive and execute commands sent by the processor 180. In addition, the touch-sensitive surface 131 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 131, the input unit 130 may further include other input devices 132. Specifically, the other input devices 132 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.
[0114] The display unit 140 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the terminal device 1200, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 140 may include a display panel 141. Optionally, the display panel 141 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 131 may cover the display panel 141. When the touch-sensitive surface 131 detects a touch operation on or near it, it is transmitted to the processor 180 to determine the type of touch event. Subsequently, the processor 180 provides corresponding visual output on the display panel 141 according to the type of touch event. Although in Figure 6 In the embodiment, the touch-sensitive surface 131 and the display panel 141 are implemented as two independent components to realize the input and output functions. However, in some embodiments, the touch-sensitive surface 131 and the display panel 141 can be integrated to realize the input and output functions.
[0115] The terminal device 1200 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 141 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 141 and / or the backlight when the terminal device 1200 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the terminal device 1200, they will not be repeated here.
[0116] The audio circuit 160, the speaker 161, and the microphone 162 can provide an audio interface between the user and the terminal device 1200. The audio circuit 160 can transmit the received audio data to the speaker 161 after converting the received audio data into an electrical signal, which is converted into a sound signal for output by the speaker 161; on the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and converted into audio data, and then the audio data is output to the processor 180 for processing, and then sent to another terminal through the RF circuit 110, or the audio data is output to the memory 120 for further processing. The audio circuit 160 may also include an earplug jack to provide communication between an external headset and the terminal device 1200.
[0117] The terminal device 1200 provides the user with wireless broadband Internet access through the transmission module 170 (eg, a Wi-Fi module). Figure 6 The transmission module 170 is shown, but it is understandable that it is not a necessary component of the terminal device 1200 and can be omitted as needed without changing the essence of the invention.
[0118] The processor 180 is the control center of the terminal device 1200. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 120, and calling data stored in the memory 120, it executes various functions of the terminal device 1200 and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 180 may include one or more processing cores; in some embodiments, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 180.
[0119] The terminal device 1200 also includes a power supply 190 for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 180 through a power management system, so that the power management system can realize functions such as managing discharge and power consumption. The power supply 190 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.
[0120] Although not shown, the terminal device 1200 may also include a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit 140 of the terminal device 1200 is a touch screen display, and the terminal device 1200 also includes a memory 120, and one or more programs, wherein one or more programs are stored in the memory 120, and are configured to be executed by one or more processors 180. One or more programs include instructions for performing the following operations:
[0121] A recognition instruction, used to control the camera to obtain an image of the object when an object placed in the refrigerator is recognized;
[0122] An extraction instruction, used to input the object image into a pre-trained information extraction model to perform an information extraction operation, and obtain the object name and the object quantity contained in the object image;
[0123] A determination instruction is used to obtain the position information of an item storage platform and the item weight of the item when a change in the load weight of the item storage platform located in the refrigerator is detected;
[0124] The binding instruction is used to bind the item name with its corresponding item quantity, location information and item weight, or to update the total quantity, location summary information and total weight of the items to which the item name is bound.
[0125] In some embodiments, the program also includes a first prompt instruction, which is used to provide a selection and entry interface and generate a first voice prompt message, prompting the user to select on the selection and entry interface whether the type of item being placed this time is a new type; if it is a new type, the time when the item is placed is recorded, and the placement time is bound to the item name corresponding to the item.
[0126] In some embodiments, the program further includes a retrieval instruction for retrieving all historical insertion times of the item for the user to view if the user selects the type of item to be inserted this time as a non-newly added type on the selection and entry interface.
[0127] In some embodiments, the program also includes a second prompt instruction for generating a second voice prompt message, or providing a display interface to display the first text prompt message to prompt the user one or more of the item name, item quantity and item location of the item currently placed.
[0128] In some embodiments, the program also includes a third prompt instruction, which is used to obtain the position information of the item storage platform, the item weight of the item, and the change value of the current load-bearing weight of the item storage platform when a change in the load-bearing weight of an item storage platform is detected; determine the status of the items currently carried by the item storage platform according to the change value, and the item status includes putting in and taking out; if the item status is taking out, control the camera to move to the position corresponding to the position information according to the position information, and obtain the item name and item quantity of the items currently carried by the item storage platform based on the camera; generate a third voice prompt message, or provide a display interface to display the second text prompt message, so as to prompt the user to select one or more of the item name, item quantity and item position of the currently taken out item.
[0129] In some embodiments, the program also includes modification instructions for providing a user correction interface for the user to determine whether to correct one or more of the item name, item quantity and item location of the item on the user correction interface based on the second voice prompt information or the first text prompt information.
[0130] In some embodiments, the program also includes a receiving instruction for receiving an information viewing request, determining the name of the item to be viewed and the viewing angle based on the information viewing request; based on the item name and the viewing angle, obtaining the item quantity, item location, item image and placement time corresponding to the item name and sending them to the sender of the information viewing request.
[0131] An embodiment of the present application also provides a terminal device.
[0132] As can be seen from the above, the embodiment of the present application provides a terminal device 1200, and the terminal device 1200 performs the following steps:
[0133] When an object placed in the refrigerator is identified, controlling the camera to obtain an image of the object;
[0134] Inputting the object image into a pre-trained information extraction model to perform an information extraction operation to obtain the object name and the object quantity contained in the object image;
[0135] When detecting that the load-bearing weight of an item storage platform located in the refrigerator changes, obtaining the position information of the item storage platform and the item weight of the item;
[0136] The item name is bound to the corresponding item quantity, location information and item weight, or the total quantity, location summary information and total item weight of the item name to which the item name is bound are updated.
[0137] An embodiment of the present application further provides a storage medium, wherein a computer program is stored in the storage medium. When the computer program is executed on a computer, the computer executes the information management method described in any of the above embodiments.
[0138] It should be noted that, for the information management method described in this application, ordinary testers in this field can understand that all or part of the process of implementing the information management method described in the embodiment of this application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as stored in the memory of a terminal device, and executed by at least one processor in the terminal device, and the execution process may include the process of the embodiment of the information management method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM, ReadOnly Memory), a random access memory (RAM, Random Access Memory), etc.
[0139] For the information management device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.
[0140] The above is a detailed introduction to the information management method, device, medium and terminal device provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. An information management method, applied to a refrigerator, characterized in that: include: When an object placed in the refrigerator is identified, controlling the camera to obtain an image of the object; Inputting the object image into a pre-trained information extraction model to perform an information extraction operation to obtain the object name and the object quantity contained in the object image; When detecting that the load-bearing weight of an item storage platform located in the refrigerator changes, obtaining the position information of the item storage platform and the item weight of the item; The item name is bound to the corresponding item quantity, location information and item weight, or the total quantity, location summary information and total item weight of the item name to which the item name is bound are updated.
2. The information management method according to claim 1, characterized in that: After the controlling camera acquires the object image of the object, the method further includes: Providing a selection entry interface and generating a first voice prompt message to prompt the user to select whether the type of the item to be put in this time is a newly added type on the selection entry interface; If it is a newly added type, the time when the item is placed is recorded, and the placement time is bound to the item name corresponding to the item.
3. The information management method according to claim 2, characterized in that: After prompting the user to select on the operation interface whether the type of the item to be placed this time is a newly added type, the method further includes: If the user selects the type of the item to be placed this time as a non-newly added type on the selection and entry interface, all historical placement times of the item are retrieved for the user to view.
4. The information management method according to claim 2, characterized in that: After binding the item name with its corresponding item quantity, location information and item weight, or updating the total quantity of items, location summary information and total weight of items to which the item name is bound, the method further includes: Generate a second voice prompt message, or provide a display interface to display the first text prompt message, to prompt the user one or more of the name of the item, the number of items, and the location of the item currently placed; A user correction interface is provided for the user to determine whether to correct one or more of the item name, item quantity and item position of the item on the user correction interface according to the second voice prompt information or the first text prompt information.
5. The information management method according to claim 4, characterized in that: After providing the user correction interface, the method further includes: In response to a first information entry operation on the user correction interface, determining whether to modify one or more of the item name, item quantity, and item location according to the first information entry operation; In response to a second information entry operation on the user correction interface, one or more of the item name, item quantity, and item location are modified according to the second information entry operation.
6. The information management method according to claim 1, characterized in that: After binding the item name with its corresponding item quantity, location information and item weight, or updating the total quantity of items, location summary information and total weight of items to which the item name is bound, the method further includes: When a change in the load-bearing weight of an item storage platform is detected, obtaining the position information of the item storage platform, the weight of the items, and the change value of the current load-bearing weight of the item storage platform; Determine the status of the items currently carried by the item storage platform according to the change value, wherein the item status includes placement and removal; If the item status is taken out, the camera is controlled to move to the position corresponding to the position information according to the position information, and the name and quantity of the items currently carried by the item storage platform are obtained based on the camera; Generate a third voice prompt message, or provide a display interface to display the second text prompt message, to prompt the user one or more of the item name, item quantity and item location of the item currently taken out.
7. The information management method according to claim 1, characterized in that: After binding the item name with its corresponding item quantity, location information and item weight, or updating the total quantity of items, location summary information and total weight of items to which the item name is bound, the method further includes: receiving an information viewing request, and determining the name of an object to be viewed and a viewing angle according to the information viewing request; According to the item name and viewing angle, the item quantity, item location, item image and placement time corresponding to the item name are obtained and sent to the sender of the information viewing request.
8. An information management device, applied to a refrigerator, characterized in that: The information management device comprises: A recognition module, used for controlling a camera to obtain an image of an object placed in the refrigerator when the object is recognized; An extraction module, used for inputting the object image into a pre-trained information extraction model to perform an information extraction operation, and obtaining the object name and the object quantity contained in the object image; a determination module, configured to obtain position information of an item storage platform and an item weight of an item when detecting that a load-bearing weight of an item storage platform located in the refrigerator changes; The binding module is used to bind the item name with its corresponding item quantity, location information and item weight, or to update the total quantity, location summary information and total weight of the items to which the item name is bound.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the information management method according to any one of claims 1 to 7.
10. A terminal device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the information management method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Method for managing storage product in refrigerator using image recognition, and refrigerator for same
CN105164481A
Method and device for updating storage information in refrigerator
CN105352260A
Intelligent recognition and management system of refrigerator
CN108759294A
Refrigerating chamber
CN109631483A
Intelligent refrigerator control method and device based on Internet of Things, medium and equipment
CN117128704A
